

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
BEFORE -THE BOARD OF PATENT APPEALS AND INTERFERENCES 



First Named 
Inventor : 

Appln. No*. : 
Filed : 
For : 



Venolia et al. 

09/421,710 

October 20, 1999^ 

METHOD AND APPARATUS FOR 
DISPLAYING SPEECH. RECOGNITION 
PROGRESS 



Docket No.: M61. 12-0144 



Appeal No. 



Group Art Unit: 2641 

Examiner RECEIVED 
Armstrong 

APR 2 4 2003 



technology Center 2 60Q 



TRANSMITTAL OF APPEAL BRIEF 
(PATENT APPLICATION - 37 C.F.R. § 192) 



Commissioner for Patents 
Washington, D.C. 20231 



I HEREBY CERTIFY THAT THIS PAPER IS 
BEING SENT BY U.S. MAIL, FIRST CLASS, 
TO THE ASSISTANT COMMISSIONER FOR 
PATENTS, WASHINGTON, D.C. 20231, THIS 

DAY OF ^f ^* ^ 




PATENT ATTORNEY 



Sir: 



Transmitted herewith in triplicate is the Appeal Brief in 
this application with respect to the Notice of Appeal filed on March 
3, 2003. 

FEE STATUS 

[ ] Small entity status under 37 C.F.R. §§ 1.9 and 1.27 

is established by a verified statement. 

FEE FOR FILING APPEAL BRIEF 

Pursuant to 37 C.F.R. 1.17(c) the fee for filing the 
Appeal Brief is $320.00. 

The Commissioner is authorized to charge any additional 
fees associated with this paper or credit any overpayment to Deposit 
Account No. 23-1123. A duplicate copy of this communication is 
enclosed. 

Respectfully submitted, 
■ WESTMAN, CHAMPLIN & KELLY, P. A. 



BEST AVAILABLE COPY 




Theodore M. Magee, Reg>^No. 39,758 
Suite 1600 - International Centre 
900 Second Avenue South 
Minneapolis, Minnesota 55402-3319 
Phone: (612) 334-3222 Fax: (612) 334-3312 



TMM:ajm: 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
BEFORE* THE BOARD OF PATENT APPEALS AND INTERFERENCES 



First Named 

Inventor : Venolia et al. 



. Appln. No-, 
Filed 
For 



09/421,710 

October 20, 1999 

METHOD AND APPARATUS FOR 
DISPLAYING SPEECH RECOGNITION 
PROGRESS 



Docket No.: M61. 12-0144 



Appeal No. 

Group Art Unit: 2641 

Armstrong ^ECEIVED 
APR 2 4 2003 



TRANSMITTAL OF APPEAL BRIEF 
(PATENT APPLICATION - 37 C.F.R. § 192) 



technology Center 2600 



Commissioner for Patents 
Washington, D.C. 20231 



Sir: 



I HEREBY CERTIFY THAT THIS PAPER IS 
BEING SENT BY U.S. MAIL, FIRST CLASS, 
TO THE ASSISTANT COMMISSIONER FOR 
PATENTS, WASHINGTON, D.C. 20231, THIS 

DAY OF f 20^3 

PATENT ATTORNEY 



Transmitted herewith in triplicate is the Appeal Brief in 
this application with respect to the Notice of Appeal filed on March 
3, 2003. 

FEE STATUS 



§§ 1.9 and 1.27 



[ ] Small entity status under 37 C.F.R, 

is established by a verified statement. 

FEE FOR FILING APPEAL BRIEF 

Pursuant to 37 C.F.R. 1.17(c) the fee for filing the 
Appeal Brief is $320.00. 

The Commissioner is authorized to charge any additional 
fees associated with this paper or credit any overpayment to Deposit 
Account No. 23-1123. A duplicate copy of this communication is 
enclosed. 

Respectfully submitted, 
WESTMAN, CHAMPLIN & KELLY, P. A. 



By 




TMM:ajm: 



Theodore M. Magee, Reg^No. 39,758 
Suite 1600 - International Centre 
900 Second Avenue South 
Minneapolis, Minnesota 55402-3319 
Phone: (612) 334-3222 Fax: (612) 334-3312 



CD 



RECEIVED 

APR 2 4 2003 

.Technology Center 2600 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
BEFORE THE BOARD OF PATENT APPEALS AND INTERFERENCES 



First Named 
Inventor : 


Venolia et al. 


Appeal No. 




Appln. No. : 


09/421,710 






Filed : 


October 20, 1999 


Group Art 


Unit: 2641 


For : 


METHOD AND APPARATUS FOR 
DISPLAYING SPEECH RECOGNITION 
PROGRESS 


Examiner: 
Armstrong 


A. 


Docket No. : 


M61. 12-0144 







BRIEF FOR APPELLANTS 



04/22/2003 YP0LITE1 00000136 09421710 

320.00 OP 

01 FC-.1A02 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
BEFORE THE BOARD OF PATENT APPEALS AND INTERFERENCES 



First Named 
Inventor : 


Venolia et al. 


nu kj \z» a -L inu i 




Appln. No. : 


09/421,710 






Filed : 


October 20, 1999 


Group Art 


Unit: 2641 


For : 


METHOD AND APPARATUS FOR 
DISPLAYING SPEECH RECOGNITION 
PROGRESS 


Examiner: 
Armstrong 


A. 


Docket No. : 


M61. 12-0144 







BRIEF FOR APPELLANT 

I HEREBY CERTIFY THAT THIS PAPER IS BEING 
SENT BY U.S. MAIL, FIRST CLASS, TO THE 
ASSISTANT COMMISSIONER OF PATENTS, 
WASHINGTON, D.C. 20231, THIS 

/^DAYOF 2<&i. 
PATENT ATTOR^Y^ 

This is an appeal of the final rejection of claims 1-33 
that was reported in the Office Action of December 3, 2002. 

REAL PARTY IN INTEREST 
Microsoft Corporation, a corporation organized under 
the laws of the state of Washington, and having offices at One 
Microsoft Way, Redmond, Washington 98052, has acquired the entire 
right, title and interest in and to the invention, the 
application, and any and all patents to be obtained therefor, as 
assignee in an Assignment recorded on Reel 010578, frame 0070. 

RELATED APPEALS AND INTERFERENCES 
Appellant knows of no appeals or interferences that 
would directly affect or be directly affected by or have a bearing 
on the Board's decision in the pending appeal. 
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STATUS OF THE CLAIMS 
Claims 1-33 are pending, rejected and appealed. 

STATUS OF AMENDMENTS 
No amendments were filed subsequent to the final 

rejection. 

SUMMARY OF INVENTION 

The invention provides a method and computer program 
for generating images of meters related to speech recognition on a 
display device 85. 

Under one embodiment of the method, an insertion marker 
214 is displayed on display 85 and a progress meter 244, 376 is 
displayed near the insertion marker. The progress meter 
quantitatively indicates the amount of progress in decoding a 
speech input. (See page 17, lines 17-20). A method of generating a 
progress meter is shown in Fig. 12 and discussed on pages 33-34. 

Under other embodiments of the present invention, a 
quantitative progress meter is combined with a volume meter as 
shown in Figs. 4G, 4H, 41, 6E, 7D, 8C and 9D. By combining the 
progress meter and the volume meter, the user is able to track the 
volume of their speech and the progress of the speech recognition 
system without substantially moving their eyes. 

Under one such embodiment , the volume meter is 
constructed according to a method shown in Fig. 11. In Fig. 11, an 
input analog value is converted into a digital value that is then 
transformed (step 459 of Fig. 11) . The transformed value is stored 
in a buffer at step 460. The value of the transformed value is 
compared to a transform maximum to determine a percentage value at 
step 4 66. This percentage value is then used to form the volume 
meter . 

In some embodiments, each speech sample is associated 
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with a separate token, such as tokens 220, which consist of a set 
of subordinate blocks such as blocks 228 and 230 of Fig. 4D. 
Tokens also include a background rectangle 232. In some 
embodiments, the progress meter is shown by converting the color 
of the background rectangle of each volume token. 

The size of the blocks in the tokens is determined at 
step 4 68. Up to three different blocks can be constructed for a 
single speech value with each block having a maximum size. The 
determination of the subordinate block sizes is discussed on page 
29, line 12 to page 31, line 5, and involves calculating a height 
ratio based on the percentage of the transformed value to the 
transform maximum and comparing the height ratio to maximum height 
ratios for each block. 

ISSUES 

Does the combination of VanBuskirk et al. (U.S. Patent 
No. 6,075,534), Tannenbaum (U.S. Patent No. 6,233,560) and Rozak 
et al. (U.S. Patent No. 5,864,815) render claims 1-3, 13-14, 17-21 
and 29 obvious under 35 U.S.C. § 103(a)? 

Does the combination of VanBuskirk et al., Tannenbaum, 
Rozak et al. and French-St. George et al. (U.S. Patent No. 
6,018,711) render claims 4-16, 20, 22-28 and 30-33 obvious under 
35 U.S.C. § 103(a)? 

GROUPING OF CLAIMS 







Appellants group the claims on appeal as follows: 
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- Claims 1, 2, 17 and 18 
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- Claim 3 
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- Claim 4 
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- Claim 7 


Group 
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- Claim 8 
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- Claim 32 






Group 


27 


- Claim 33 







ARGUMENT 
Group 1 - Claims 1, 2, 17 and 18 

Independent claims 1 and 17 and dependent claims 2 and 
18 were rejected under 35 U.S.C. § 103(a) as being unpatentable 
over VanBuskirk et al. (U.S. Patent No. 6,075,537, hereinafter 
VanBuskirk) in view of Tannenbaum (U.S. Patent No. 6,233,560) and 
Rozak et al. (U.S. Patent No. 5,864,815, hereinafter Rozak) . 

Independent claim 1 provides a method of displaying 
images on a display device. The method includes displaying an 
insertion marker at an insertion area on a display and displaying 
a progress meter near the insertion area based on the location of 
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the insertion marker. The progress meter quantitatively indicates 
the amount of progress in decoding a speech input. 

Independent claim 17 provides a computer program having 
a least one insertion point marker that indicates a location on 
the display where a user desires to provide input. The computer 
program also includes a speech recognition routine and a meter 
generation routine that displays a progress meter near an 
insertion point based on the insertion point marker. The progress 
meter is quantitatively indicative of the amount of speech that 
has been decoded by the speech recognition routine. 

The combination of VanBuskirk, Tannenbaum and Rozak 
does not show or suggest the invention of claim 1 or claim 17 
because none of these references show a progress meter that 
quantitatively indicates the amount of progress in decoding a 
speech input. In addition, none of these references show or 
suggest the ability to place such a progress meter at an insertion 
area. 

VanBuskirk describes a volume tracking window for a 
speech recognition system. Under VanBuskirk, the detected volume 
of a speech signal is represented in a window by changing the 
color of the entire window or by moving a colored bar horizontally 
to show the current volume. The volume tracking window in 
VanBuskirk may be a "floating window", however, VanBuskirk does 
not show or suggest that the volume tracking window should be 
placed near, an insertion marker. In addition, VanBuskirk does not 
show or suggest a progress meter that shows the amount of progress 
in decoding an input speech signal. 

Tannenbaum discloses a speech recognition interface in 
which fully recognized phrases or commands are displayed in a box. 
Under Tannenbaum, the recognized phrase or command is used to 
determine where to position the box on the display. Tannenbaum 
does not disclose displaying a progress meter that indicates the 
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amount of progress in recognizing a speech signal. In addition, 
because Tannenbaum requires the recognition of the command before 
it can determine where to place the recognition result, its 
teachings could not be used to position a recognition progress 
meter since the commands would not be known during the period of 
time when the progress meter needs to be displayed. Thus, under 
Tannenbaum, it would not be possible to place a progress meter 
near an insertion area since the command is unknown until 
recognition is complete. 

Rozak discloses a graphical notification that indicates 
the processing state of a speech recognizer. The graphical 
notification provides an icon showing whether the speech 
recognizer is currently processing audio inputs. However, Rozak 
does not teach or suggest a progress meter that quantitatively 
indicates the amount of progress in decoding a speech input. Rozak 
also does not suggest placing a progress meter near an insertion 
area. 

In the Final Office Action, the Examiner asserted that 
even though no single reference showed a quantitative progress 
meter, it would have been obvious to combine the quantitative 
aspect of the volume meter in VanBuskirk with the non-quantitative 
status indicator of Rozak to produce a quantitative meter that is 
indicative of the amount of speech recognition that has been 
performed. Appellants dispute this assertion. 

First, there is no suggestion in Rozak or VanBuskirk 
for taking the quantitative attribute of the volume meter provided 
by VanBuskirk and combining it with the state information shown by 
Rozak. As stated by the Federal Circuit in In re Lee , 277 F.3d 
1338, 61 U.S.P.Q.2d 1430, 1432 (Fed. Cir. 2002 ), "particular 
findings must be made as to the reason the skilled artisan, with 
no knowledge of the claimed invention, would have selected these 
components for combination in the manner claimed" (citing In re 
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Rouffet, 149 F.3d 1350, 1359, 47 U.S.P.Q.2d 1453, 1459 (Fed. Cir. 
1998)). However, in VanBuskirk and Rozak there are no such 
suggestions. VanBuskirk does not indicate that the quantitative 
aspect of its volume meter should be applied to other types of 
display meters and Rozak does not indicate that there are any 
problems with its non-quantitative meter that would require- it to 
be changed. 

In fact, the selective combination of these aspects of 
VanBuskirk and Rozak was only made by the Examiner after reading 
the blueprint provided by the present application. Without the 
description of the present invention provided by this application, 
neither the Examiner nor those skilled in the art would have 
thought to combine the quantitative aspect of the volume meter of 
VanBuskirk with the basic status display of Rozak. The Examiner is 
clearly using hindsight in forcing the combination of these two 
references to produce the invention described in the present 
application. It is this type of hindsight that the Federal Circuit, 
has attempted to limit by requiring that the Examiner find some 
suggestion for selecting particular components for combination. 

In addition, Appellants note that this selective 
combination does not teach those skilled in the art how to 
practice the present invention. In particular, neither VanBuskirk 
nor Rozak teach a manner in which a progress meter that 
quantitatively indicates the amount of progress in decoding a 
speech input could be displayed. In VanBuskirk, the quantitative 
amount of the volume is easily displayed by selecting a maximum 
value to be expected, and displaying a ribbon based on the ratio 
between the current volume and the maximum volume. However, 
neither VanBuskirk nor Rozak show a way to set a maximum value to 
indicate the progress of decoding a speech input. In particular, 
neither reference shows how to determine how large to make the 
progress meter at any point in time or how to change the progress 
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meter over time so that it quantitatively shows the progress in 
decoding a speech input . Without such teachings, it is not clear 
how such a progress meter could be formed from VanBuskirk and 
Rozak. 

Since there is no suggestion to combine the selected 
elements of VanBuskirk and Rozak, and since the references do not 
teach how to generate a quantitative progress meter, it would not 
be obvious to combine the selected elements in VanBuskirk and 
Rozak to form the invention of claims 1, 2, 17 and 18. 

In addition, the combination of the cited references do 
not show or suggest the ability to place a progress meter at an 
insertion area designated by an insertion marker. In the Final 
Office Action, it was asserted that Tannenbaum teaches this aspect 
of the invention because it teaches that recognized commands 
should be displayed at a location functionally related to the 
analyzed contents and context of the voice input. However, the 
teachings of Tannenbaum could not be applied to a progress meter. 
In particular, Tannenbaum 1 s system requires that the input be 
analyzed to determine which command has been spoken. See column 2, 
lines 40-47 and column 7, lines 7-24. Note in particular that the 
system does not know if the user has issued a command to open a 
file or move a cursor until it has decoded the word spoken by the 
user. Without that information, Tannenbaum does not know whether 
to place the feedback near the tool bar or near the cursor. Thus, 
Tannenbaum must wait for the recognition to be performed before 
placing its feedback box on the screen. 

However, waiting to place the feedback box until after 
the recognition is complete defeats the purpose of displaying a 
progress meter indicating the progress of recognition. In 
particular, since Tannenbaum must wait for the full recognition to 
finish, it is impossible for Tannenbaum' s system to be used to 
place a progress meter that must be displayed before the 
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recognition is completed. 

In light of the fact that none of the cited references 
provide a system that allows a progress meter to be positioned at 
an insertion marker and because there is no suggestion in the 
cited references for displaying a quantitative progress meter or 
suggestions of how a progress meter could be constructed, the 
invention of claims 1, 2, 17 and 18 is patentable over VanBuskirk, 
Rozak and Tannenbaum. 

Group 2 - Claim 3 

Dependent claim 3 depends from claim 1. It does not 
stand or fall with claim 1 because it includes an additional 
limitation that is not shown or suggested in the combination of 
VanBuskirk, Rozak and Tannenbaum. 

In claim 3, a user 1 s speech input is converted into an 
analog speech signal and that signal is converted into at least 
one digital speech value. The at least one digital speech value ; is 
then transformed into coordinates for at least one shape on the 
display which is positioned near the progress meter. 

Claim 3 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, and Tannenbaum. 

The combination of VanBuskirk, Rozak and Tannenbaum 
does not show or suggest a display that includes both a progress 
meter and a shape that is formed by transforming a digital speech 
value into coordinates. 

In the Final Office Action, it was asserted that 
VanBuskirk teaches that a multi-function graphical user interface 
should be shown in as small a space as possible and that it would 
therefore be obvious to display a volume meter near a progress 
meter. Applicants dispute this assertion. 

First, Appellants note that none of these references 
show or suggest a progress meter that quantitatively shows the 
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amount' of progress in decoding a speech * signal. Thus, those 
skilled in the art would first have to create such a progress 
meter from these references without even a suggestion in any of 
the references that such a progress meter should be constructed. 

After the progress meter is constructed, those skilled 
in the art still would not have produced the invention of claim 3. 
To produce the invention of claim 3, those skilled in the art 
would have to take the further step of combining the progress 
meter with another shape formed by transforming a digital speech 
value. However, there is no suggestion in the art for combining a 
progress meter with such a shape in the cited art. 

Although VanBuskirk does teach that different types of 
information can be conveyed in the same graphical interface, it 
does not show or suggest that a shape formed by transforming a 
digital speech value should be combined with a progress meter. 

Since none of the cited references show or suggest a 
progress meter, the combination of references cannot show or 
suggest the combination of a progress meter and a shape generated 
from a digital speech value. As such, claim 3 is patentable over 
VanBuskirk, Rozak and Tannenbaum. 

Group 3 - Claim 4 
Claim 4 depends from claim 3. Claim 4 does not stand or 
fall with claim 3 because it includes an additional limitation 
that is not found in the combination of VanBuskirk, Rozak and 
Tannenbaum. 

In claim 4, the step of transforming at lest one 
digital speech value into coordinates includes applying a 
mathematical function to the at least one digital speech value to 
produce a transform value and using the transform value to 
identify coordinates for the shape on the display. 

Claim 4 was rejected under 35 U.S.C. § 103(a) as being 
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unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George et al. (U.S. Patent No. 6,018,711, hereinafter French-St. 
George) . 

French-St. George discloses an animation that indicates 
the amount of time the user has left in which to provide speech 
input to a speech recognizer. French-St. George does not suggest 
that this animation should be placed near an insertion point and 
does not show or suggest a progress meter that indicates the 
amount of progress in decoding an input speech segment. In 
addition, French-St. George does not suggest identifying 
coordinates for at least one shape on the display by applying a 
mathematical function to at least one digital speech value to 
produce a transform value. 

In fact, none of the references in the combination of 
VanBuskirk, Rozak, Tannenbaum, and French-St. George show or 
suggest applying a mathematical function to at least one digital 
speech value to produce a transform value and then using that 
transform value to identify coordinates for at least one shape on 
a display. In rejecting claim 4, the Examiner never asserted that 
any of the references show a mathematical function applied to at 
least one digital speech value to produce a transform value. As 
such, the invention of claim 4 is not shown or suggested in the 
combination of the references cited by the Examiner and claim 4 is 
therefore patentable over VanBuskirk, Rozak, Tannenbaum and 
French-St . George . 

Group 4 - Claim 5 
Claim 5 depends from 4. Claim 5 does not stand or fall 
with claim 4 because it includes a further limitation wherein 
applying a mathematical function comprises taking the logarithm 
value of at least one digital value. This additional limitation is 
not shown or suggested in the combination of VanBuskirk, Rozak, 
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Tannenbaum and French-St. George. 

Claim 5 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. However, none of the cited references show or suggest 
taking a logarithm of a digital speech value. Furthermore, the 
Examiner has not asserted that any particular section of any of 
these references shows the taking of a logarithm of a digital 
speech value. 

Since none of the references show or suggest taking the 
logarithm of a speech value, claim 5 is patentable over the 
combination of VanBuskirk, Rozak, Tannenbaum and French-St. 
George. 

Group 5 - Claim 6 
Claim 6 depends from 4 and includes a further 
limitation wherein the mathematical function comprises taking the 
square root of at least one digital value. Claim 6 does not stand 
or fall with claim 4 because the additional limitation of taking a 
square root of a digital speech value is not shown in the cited 
references . 

Claim 6 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. 

None of the cited references show or suggest taking the 
square root of at least one digital speech value to produce a 
transform value that is then used to identify the coordinates of 
at least one shape on the display. In addition, no section of the 
references has been cited as showing the application of a square 
root function to a digital speech value. 

As such, claim 6 is patentable over the combination of 
VanBuskirk, Rozak, Tannenbaum and French-St. George. 
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Group 6 - Claim 7 

Claim 7 depends from claim 4 but stands apart from 
claim 4 because it includes a further limitation wherein the shape 
formed from the transformed value is a base rectangle and 
determining the coordinates of the base rectangle includes a 
number of steps that are not shown in the combination of 
VanBuskirk, Rozak, Tannenbaum and French-St. George. 

In claim 7, coordinates of the base rectangle are 
determined by determining a base point for the base rectangle on 
the display, accessing a stored rectangle width, and accessing a 
maximum transform value. The transform value formed from the 
digital speech value is divided by the maximum transform value to 
produce a transform ratio. A height is calculated based in part on 
the transform ratio. The coordinates of the base rectangle are 
then calculated based on the base point, the stored rectangle 
width, and the calculated height. 

Claim 7 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. However, none of the cited references show or suggest the 
steps of claim 7. In particular, none of the cited references show 
or suggest accessing a maximum transform value, dividing a 
transform value produced from a digital speech value by the 
maximum transform value, or using the resulting ratio to calculate 
coordinates for a base rectangle. 

Since none of these steps are shown in any of the 
references, the invention of claim 7 is patentable over 
VanBuskirk, Rozak, Tannenbaum and French-St. George. 

Group 7 - Claim 8 
Claim 8 depends from claim 7 and includes a further 
limitation where determining a calculated height includes 
determining if the transform ratio is greater than a maximum 
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height ratio for the base rectangle and if it is, performing a 
further step of multiplying the maximum height ratio for the base 
rectangle by the full meter height to produce the calculated 
height. Since this limitation is not shown in the combination of 
VanBuskirk, Rozak, Tannenbaum, and French-St. George, claim 8 does 
not stand or fall with claim 7. 

Claim 8 was re j ected under 35 U.S.C. § 103(a) as being 
unpatentable over the combination of VanBuskirk, Rozak, Tannenbaum 
and French-St. George. However, none of the cited references show 
or suggest the steps of claim 8. In particular, none of the 
references show a step of comparing a transform ratio to a maximum 
height ratio for a base rectangle to determine which is larger. In 
addition, none of the references show or suggest performing a 
further step of multiplying the maximum height ratio for the base 
rectangle by a full meter height if the transform ratio is greater 
than the maximum height ratio. 

Since none of the cited references show these steps, 
the combination of these references does not show or suggest these 
steps. As such, claim 8 is patentable over the combination of 
VanBuskirk, Rozak, Tannenbaum and French-St. George. 

Group 8 - Claim 9 

Claim 9 depends from claim 8 and was re j ected under 35 
U.S.C. § 103(a) as being unpatentable over VanBuskirk, Rozak, 
Tannenbaum and French St . George . Claim 9 does not stand or fall 
with claim 8 because it includes an additional limitation to 
determine the coordinates of a second rectangle based on the 
coordinates of a base rectangle, which is not shown or suggested 
in the cited art. 

In claim 9, the maximum height ratio for the base 
rectangle is subtracted from the transform ratio to produce an 
excess ratio . A second rectangle height is then determined based 
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in part on the excess ratio. The coordinates of a second rectangle 
are then calculated based on the coordinates of the base 
rectangle, a stored rectangle width, and the second rectangle 
height. 

None of the cited references show a step of subtracting 
a maximum height ratio from a transform ratio or a step of using 
the resulting difference to determine the height of a second 
rectangle. As such, the combination of references does not show 
the invention of claim 9. Claim 9 is therefore patentable over 
VanBuskirk, Rozak, Tannenbaum and French-St. George. 

Group 9 - Claim 10 

Claim 10 depends from claim 9 and includes a further 
limitation wherein determining the second rectangle height 
involves comparing the excess ratio to a maximum height ratio for 
the second rectangle. Since this additional limitation is not 
shown in the references used to reject claim 10, claim 10 does not 
stand or fall with claim 9. 

Claim 10 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. However, no particular section of any of these references 
was cited by the Examiner to support the rejection of claim 10 and 
in fact, no section of these references shows a step of 
determining a rectangle height for a second rectangle. As such, 
claim 10 is patentable over VanBuskirk, Rozak, Tannenbaum and 
French-St. George. 

Group 10 - Claim 11 
Claim 11 depends from claim 10 but does not stand or 
fall with claim 10 because it includes an additional limitation to 
determining the coordinates of a third rectangle, which is not 
shown or suggested in the cited art. 
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Claim 11 was rejected under 35 U.S.C. § 103(a) as being 
obvious from VanBuskirk, Rozak, Tannenbaum and French-St. George. 

Under claim 11, the maximum height ratio for the second 
rectangle is subtracted from the excess ratio to produce a 
remainder ratio. A third rectangle height is then determined by 
multiplying the remainder ratio by the full meter height. The 
coordinates of a third rectangle is then determined from the third 
rectangle height. 

None of the references in the combination of 
VanBuskirk, Rozak, Tannenbaum and French-St. George shows or 
suggests determining the coordinates of a third rectangle. In the 
Final Office Action, no portion of any of these references was 
cited as disclosing the calculation of a third rectangle. As such, 
the invention of claim 11 is patentable over the combination of 
VanBuskirk, Rozak, Tannenbaum and French-St. George. 

Group 11 - Claim 12 

Claim 12 depends from claim 7. Claim 12 does not stand 
or fall with claim 7 because it includes a further limitation of 
calculating the coordinates of a background rectangle that is not 
found in the cited references. 

Claim 12 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George . 

Claim 12 includes a limitation to calculating the 
coordinates of a background rectangle where the background 
rectangle appears somewhere between the base rectangle and a point 
at a full meter height above a bottom edge of the base rectangle. 
None of the cited references disclose calculating the coordinates 
of such a background rectangle. In addition, the Final Office 
Action did not cite any portion of these references as showing 
such a background rectangle. As such, the invention of claim 12 is 
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patentable over the combination of VanBuskirk, Rozak, Tannenbaum 
and French-St. George. 

Group 12 - Claim 13 

Claim 13 depends from independent claim 1. Claim 13 
does not stand or fall with claim 1 because it includes a further 
limitation that is not shown or suggested in the cited references. 

Claim 13 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak and Tannenbaum and as being 
unpatentable over the combination of VanBuskirk, Rozak, Tannenbaum 
and French-St. George. 

In claim 13, the progress meter of claim 1 is displayed 
by dividing the speech input into frames, and decoding at least 
one of the frames of the speech input into a sub-word unit. A 
frame number for the last frame to be decoded is divided by the 
total number of frames to produce a decode ratio. The progress 
meter is then displayed based on this decode ratio. 

None of the cited references show or suggest a step of 
dividing the frame number of a last frame to be decoded by the 
total number of frames to produce a decode ratio. Further, none of 
the references show or suggest using a decode ratio to display a 
progress meter. As such, claim 13 is patentable over the 
combination of VanBuskirk, Rozak and Tannenbaum and the 
combination of VanBuskirk, Rozak, Tannenbaum and French-St. 
George . 

Group 13 - Claim 14 

Claim 14 depends from claim 13. Claim 14 does not stand 
or fall with claim 13 because it includes a further limitation 
that is not shown in the cited references. 

Like claim 13, claim 14 was rejected under 35 U.S.C. § 
103(a) as being unpatentable over VanBuskirk, Rozak and Tannenbaum 
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and as being unpatentable over the combination of VanBuskirk, 
Rozak, Tannenbaum and French-St. George. 

Claim 14 includes limitations to multiplying the decode 
ratio by a full meter width to determine a progress width and 
calculating the coordinates of a progress rectangle based on the 
progress width, a stored meter height and a base point on the 
display. None of the cited references show or suggest multiplying 
a decode ratio by a full meter width, nor do they show or suggest 
calculating the coordinates of a progress rectangle based on the 
result of such a multiplication. Since none of the references show 
any of these steps, their combination does not show or suggest the 
invention of claim 14, and claim 14 is therefore patentable over 
VanBuskirk, Rozak and Tannenbaum and the combination of 
VanBuskirk, Rozak, Tannenbaum and French-St. George. 

Group 14 - Claim 15 

Claim 15 depends from claim 12. It does not stand or 
fall with claim 12 because it includes further limitations that 
are not found in the cited references. 

Claim 15 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. However, no particular section of any of these references 
was cited as showing or suggesting any of the elements of claim 
15. 

In claim- 15, the speech input is divided into frames 
and at least one frame of the speech is decoded into a sub-word 
unit. A frame number for the last frame to be decoded is divided 
by the total number of frames to produce a decode ratio. A 
progress meter is then displayed based on the decode ratio by 
changing the color of at least one background rectangle. 

None of the cited references discuss dividing a frame 
number of the last frame to be decoded by the total number of 
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frames to produce a decode ratio. In addition, none of the 
references discuss using the decode ratio to change the color of 
at least one background rectangle and thereby display a progress 
meter. 

Since none of the references discuss these steps, their 
combination does not show or suggest the steps of claim 15. As 
such, claim 15 is patentable over the combination of VanBuskirk, 
Rozak, Tannenbaum and French-St. George. 

Group 15 - Claim 16 

Claim 16 depends from claim 15 but does not stand or 
fall with claim 15 because it includes a further limitation that 
is not shown or suggested in the cited art. 

Claim 16 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George . 

In claim 16, the step of displaying a progress meter in 
claim 15 is defined as multiplying the decode ratio by a full 
meter width to produce a progress width and dividing the progress 
width by a rectangle width that is indicative of the width of each 
background rectangle . This produces a rectangle count . The color, 
of a number of background rectangles is then changed where the 
number of background rectangles is equal to the rectangle count. 

None of the cited references show or suggest the steps 
of dividing a progress width by a rectangle width to get a 
rectangle count or of changing the color of a number of background 
rectangles equal to the rectangle count. As such, the combination 
of references does not show these steps. 

Since none of the references show the steps of claims 
16, the invention of claim 16 is patentable over VanBuskirk, 
Rozak, Tannenbaum and French-St. George. 
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Group 16 - Claim 19 

Claim 19 depends from claim 17 but does not stand or 
fall with claim 17 because it includes an additional limitation 
that is not found in the cited art. 

In particular, claim 19 includes a limitation where a 
meter generation routine further comprises a transform routine 
that transforms a digital value into a set of coordinates for a 
shape on the display. The digital value is indicative of the 
magnitude of a portion of the speech signal. Thus, the meter 
generation routine of claim 19 is able to generate a progress 
meter that quantitatively indicates the progress of decoding while 
at the same time providing a shape on the display that indicates 
the magnitude of the speech signal. 

Claim 19 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak and Tannenbaum. 

In the Final Office Action, it was noted that none of 
these references specifically teach displaying a volume meter 
close to a progress meter . However, it was asserted that since 
VanBuskirk teaches that a multi-function graphical user interface 
should supply information in the smallest possible space, it would 
be obvious to display a volume meter close to a decoding progress 
meter. Appellants dispute this assertion. 

Appellants note that none of these references show or 
suggest a progress meter that quantitatively shows the amount of 
progress in decoding a speech signal. Thus, those skilled in the 
art would first have to create such a progress meter without any 
suggestion from these references to do so. 

After the progress meter is constructed, those skilled 
in the art still would not have produced the invention of claim 
19. To produce the invention of this claim, those skilled in the 
art would have to take a further step of combining the progress 
meter with a shape that indicates the magnitude of a speech value. 



-22- 



However, there is no ' suggestion in the art for making such a 
combination. 

Although VanBuskirk does teach that different types of 
information can be conveyed in the same graphical interface, it 
does not show or suggest that a volume meter should be combined 
with a progress meter. In addition, none of the references 
disclose how a progress meter could be displayed near a volume 
meter without confusing the user as to what the meters are 
conveying. Thus, there is no teaching or suggestion in any of the 
cited art for how to place a volume meter near a progress meter. 

In light of the fact that none of the references show a 
progress meter that shows the amount of progress in decoding a 
speech input and that none of the references show or suggest 
combing such a progress meter with a shape representing the 
magnitude of a speech value, the invention of claim 19 is 
patentable over the combination of VanBuskirk, Rozak and 
Tannenbaum. 

Group 17 - Claim 20 

Claim 20 depends from claim 17 but does not stand or 
fall with claim 17 because it was rejected based on a different 
combination of references from claim 17. In particular, claim 20 
was rejected under 35 U.S.C. § 103(a) as being unpatentable over 
VanBuskirk, Rozak, Tannenbaum and French-St. George. 

Claim 20 includes a further limitation wherein the 
speech recognition routine decodes the speech signal into a set of 
subwords . 

As discussed above for claim 17, the combination of 
VanBuskirk, Rozak, and Tannenbaum does not show a quantitative 
progress meter and does not suggest how a quantitative progress 
meter can be constructed. 

French-St. George also fails to show a quantitative 
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progress meter that shows the amount of progress in decoding a 
speech signal- French-St. George only shows an animation related 
to the amount of time left for a user to provide speech to a 
speech recognition system. It does not discuss the progress of 
decoding a speech signal. 

Since French-St. George does not disclose a 
quantitative progress meter that indicates the amount of progress 
in decoding a speech signal and none of VanBuskirk, Rozak, and 
Tannenbaum show such a meter, the invention of claim 20 is 
patentable over the combination of VanBuskirk, Rozak, Tannenbaum, 
and French-St. George. 

Group 18 - Claims 21 and 29 

Independent claims 21 and 29 stand apart from 
independent claims 1 and 17 because they include a combination of 
limitations that are not found in independent claims 1 and 17. 

Independent claim 21 is directed to a method in a 
computer system that displays a volume meter that is indicative of 
the magnitude of at least a portion of a speech input. The volume 
meter is displayed near a progress meter, where the progress meter 
quantitatively indicates the amount of progress in decoding the 
speech signal. 

Independent claim 29 provides a computer program 
designed to operate in a computer system having a display. The 
program includes a volume meter portion that is capable of 
displaying a volume meter, a speech recognition portion that is 
capable of converting human speech into a set of sub-words, and a 
progress meter portion that is capable of generating a progress 
meter that is quantitatively indicative of the amount of progress 
in converting the human speech signal into sub-words. 

Claims 21 and 29 were rejected under 35 U.S.C. § 103(a) 
as being unpatentable over VanBuskirk, Rozak, and Tannenbaum. 
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In the Final Office Action, " the Examiner stated that 
none of VanBuskirk, Tannenbaum or Rozak specifically teaches 
displaying a volume meter close to a progress meter. However, the 
Examiner asserted that VanBuskirk teaches that multiple functions 
should be placed in a single graphical user interface so that the 
interface is as small as possible and that because of this, it 
would be obvious to combine a volume meter with a progress meter. 
Appellants dispute this assertion. 

Appellants note that none of these references show or 
suggest a progress meter that quantitatively shows the amount of 
progress in decoding a speech signal. Thus, those skilled in the 
art would first have to create such a progress meter without even 
a suggestion in the references that such a progress meter should 
be constructed and without any teachings as to how to construct 
such a meter. 

After the progress meter is constructed, those skilled 
in the art still would not have produced the invention of claims 
21 and 29. To produce the invention of those claims, those skilled 
in the art would have to take the further step of combining the 
progress meter with a volume meter so that the two meters are 
displayed close together. Once again, there is no suggestion in 
the art for combining a volume meter with a progress meter. 

Although VanBuskirk does teach that different types of 
information can be conveyed in the same graphical interface, it 
does not show or suggest that a volume meter should be combined 
with a progress meter. In addition, none of the references 
disclose how a progress meter could be displayed near a volume 
meter without confusing the user as to what the meters are 
conveying. Thus, there is no teaching or suggestion in any of the 
cited art for how to place a volume meter near a progress meter. 

In light of the fact that none of the references show 
or suggest a quantitative progress meter and therefore are 
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incapable * of showing or suggesting that such a progress meter 
should be placed near a volume meter, the combination of 
references does not show or suggest the invention of claims 21 and 
29. As such f claims 21 and 29 are patentable over VanBuskirk, 
Rozak and Tannenbaum. 

Group 19 - Claims 22 and 23 

Claim 22 depends from claim 21 and claim 23 depends 
from claim 22. Claims 22 and 23 do not stand or fall with claim 21 
because they include a further limitation that is not shown in the 
cited art and because they were rejected based on a different 
combination of art. 

Claim 22 includes a further limitation wherein 
displaying a volume meter includes storing digital values 
representing the magnitudes of different portions of the speech 
signal. A separate token is then displayed for each separate 
digital value. 

Claims 22 and 23 were rejected under 35 U.S.C. § 103(a) 
as being unpatentable over VanBuskirk, Rozak, Tannenbaum and 
French-St. George. No portions of any of these references were 
cited as showing a step of displaying a separate token for 
separate digital values representing the magnitude of a speech 
signal . 

In fact, none of the references show or suggest 
displaying a separate token for separate digital values of a 
speech signal. Since none of the cited references show displaying 
a separate token, claims 22 and 23 are patentable over VanBuskirk, 
Rozak, Tannenbaum and French-St. George. 

Group 20 - Claim 24 
Claim 24 depends from claim 23. It does not stand or 
fall with claim 23 because it includes an additional limitation 
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that is not shown in the cited references. 

In claim 24 , a digital value representing the magnitude 
of a portion of a speech signal is transformed into a transform 
value. This transform value is then divided by a maximum meter 
value to produce a meter ratio. The height of a meter portion of a 
token is then determined using the meter ratio and the full meter 
height . 

Claim 24 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. However, no section of any of these references was cited 
as showing a step of transforming a digital value into a transform 
value and then dividing the transform value by a maximum meter 
value . 

In fact, none of the references show or suggest these 
steps. As such, the invention of claim 24 is patentable over 
VanBuskirk, Rozak, Tannenbaum and French-St. George. 

Group 21 - Claim 25 

Claim 25 depends from claim 24 and includes a further 
limitation wherein determining the height comprises multiplying 
the meter ratio by the full meter height. Claim 25 does not stand 
or fall with claim 24 because this additional limitation is not 
found in the cited references. 

Claim 25 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. However, none of these references show or suggest 
multiplying a meter ratio by a full meter height to determine the 
height of a meter portion of a token. As such, the invention of 
claim 25 is patentable over VanBuskirk, Rozak, Tannenbaum and 
French-St . George . 
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Group 22 - Claim 2 6 
Claim 26 depends from claim 24 and includes a further 
limitation wherein determining the height of a meter portion of a 
token includes performing a series of steps that are not shown in 
the cited references. As such, claim 26 does not stand or fall 
with claim 24. 

Claim 26 includes steps of determining if the meter 
ratio is greater than a base ratio and if the meter ratio is 
greater, multiplying the base ratio by the full meter height to 
determine the height of a base block while subtracting the base 
ratio from the meter ratio to produce an excess ratio. This excess 
ratio is then used with the full meter height to determine the 
height of a second block of the meter portion. 

Claim 26 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. However, no particular portion of any of these references 
was cited to support this rejection. 

None of the cited references mention any of the steps 
of comparing a meter ratio to a base ratio, multiplying a base 
ratio by a full meter height, subtracting a base ratio from a 
meter ratio to produce an excess ratio, or using an excess ratio 
and the full meter height to determine the height of a second 
block. As such, the combination of references does not show these 
steps. In light of this, claim 26 is patentable over the 
combination of VanBuskirk, Rozak, Tannenbaum and French-St. 
George . 

Group 23 - Claim 27 

Claim 27 depends from claim 26 but does not stand or 
fall with claim 26 because it includes further steps that are not 
shown in the cited references. 

In particular, claim 27 includes steps of comparing the 
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excess ratio to an intermediate ratio to determine if it exceeds 
the intermediate ratio and if it does exceed the intermediate 
ratio multiplying the intermediate ratio by the full meter height 
to produce the height of the second block. The intermediate ratio 
is then subtracted from the excess ratio to produce a remainder 
ratio. This is multiplied by the full meter height to produce the 
height for a top block of the meter portion. 

Claim 27 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. However, none of these references show or suggest any of 
the steps found in claim 27. As such, claim 27 is patentable over 
the cited references. 

Group 24 - Claim 28 

Claim 28 depends from claim' 24 but does not stand or 
fall with claim 24 because it includes further limitations that 
are not found in the cited references. 

In particular, claim 28 includes steps of dividing a 
number for a last decoded frame by a total number of frames to 
produce a progress ratio and multiplying the progress ratio by a 
full meter width to produce a progress width. The progress width 
is then divided by a token width to produce an affected number of 
tokens. For each of the affected number of tokens, the color of at 
least a portion of the token is changed so that it is different 
from the color of other tokens. Thus, claim 28 displays the 
progress meter by modifying the tokens used for the volume meter. 

Claim 28 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. No portions of any of these references were cited as 
showing any of the steps of claim 28. 

In fact, none of these references show any of these 
steps. As such, the combination of these references does not show 
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or suggest the steps of claim 28. Therefore, claim 28 is 
patentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. 

Group 25 - Claims 30 and 31 

Claim 30 depends from claim 2 9 and claim 31 depends 
from claim 30. Claims 30 and 31 do not stand or fall with claim 
29 because they were rejected using a different combination of 
references than those used to reject claim 29. 

Claims 30 and 31 were rejected under 35 U.S.C. § 103(a) 
as being unpatentable over VanBuskirk, Rozak, Tannenbaum and 
French-St. George. Since claims 30 and 31 depend from claim 29, 
they include the limitations to displaying a progress meter near a 
volume meter that are found in claim 29. 

As discussed above for claim 29, the combination of 
VanBuskirk, Rozak, and Tannenbaum does not show or suggest a 
quantitative progress meter in combination with a volume meter. 
Similarly, French-St . George does not show or suggest a 
quantitative progress meter that shows the amount of progress in 
decoding a speech signal and does not show or suggest combining 
such a progress meter with a volume meter. As such, the 
combination of VanBuskirk, Rozak, Tannenbaum and French-St .George 
does not show the invention of claim 29 or claims 30 and 31, which 
depend from claim 29. Therefore, claims 30 and 31 are patentable 
over VanBuskirk, Rozak, Tannenbaum and French-St . George . 

Group 26 - Claim 32 

Claim 32 depends from claim 31 but does not stand or 
fall with claim 31 because it includes a further limitation that 
is not shown or suggested in the cited art. 

In particular, claim 32 includes components for taking 
a frame number representing the last frame of speech that was 
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decoded by a speech recognition system and dividing it by a total 
frame number representing the total number of frames found in the 
speech signal. It also includes a component for determining a 
maximum dimension for the progress meter and code for multiplying 
the progress ratio by the maximum dimension to produce a progress 
dimension. 

Claim 32 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. In the Final Office Action, no particular sections of any 
of these references were cited as showing the functions described 
in claim 32. 

In fact, none of the cited references show or suggest 
any of the functions performed in claim 32. As such, the 
combination of these references does not show or suggest the 
invention of claim 32. Claim 32 is therefore patentable over 
VanBuskirk, Rozak, Tannenbaum and French-St. George. 

Group 27 - Claim 33 

Claim 33 depends from claim 32 but does not stand or 
fall with claim 32 because it includes a further limitation that 
is not found in the cited references. In particular, claim 33 
includes a limitation wherein volume token program code generates 
a volume token based in part on the progress dimension. 

Claim 33 was rejected under 35 U.S.C. § 103(a) as being 
unpatentable over VanBuskirk, Rozak, Tannenbaum and French-St. 
George. No particular portions of those references were cited to 
support the rejection of claim 33. In particular, no portions were 
cited as showing the generation of a volume token based in part on 
a progress dimension. 

The invention of claim 33 is not shown or suggested in 
any of the cited references. In particular, none of the references 
show or suggest that a volume token should be based in part on a 
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progress dimension. As such, the invention of claim 33 is 
patentable over the combination of VanBuskirk, Rozak, Tannenbaum 
and French-St. George. 



CONCLUSION 

In conclusion, claims 1-33 are not shown or suggested 
in the combination of VanBuskirk, Rozak, Tannenbaum and French-St. 
George. Appellants therefore request reversal of the Examiner's 
rejection of claims 1-33. 

Respectfully submitted, 
WESTMAN, CHAMPLIN & KELLY, P. A. 



By: 




Theodore M. Magee, Rfeg. No. 39,758 
Suite 1600 - International Centre 
900 Second Avenue South 
Minneapolis, Minnesota 55402-3319 
Phone: (612) 334-3222 Fax: (612) 334-3312 
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Appendix A 

Claims On Appeal : 

1. A method in a computer system for generating images on 
a display device, the method comprising: 

displaying an insertion marker at an insertion area on 
a display, the insertion area representing the 
location at which the user desires to provide 
input; and 

displaying a progress meter near the insertion area 
based on the location of the insertion marker, the 
progress meter quantitatively indicative of the 
amount of progress in decoding a speech input. 

2. The method of claim 1 further comprising: 

receiving input from the user indicating that a 

microphone is to be activated; 
activating the microphone; and 

displaying an indication that the microphone is active 
near the progress meter. 

3. The method of claim 1 further comprising: 
converting a user's speech input into an analog speech 

signal; 

converting the analog speech signal into at least one 
digital speech value; and 

transforming the at least one digital speech value into 
coordinates for at least one shape on the display 
positioned near the progress meter. 

4. The method of claim 3 wherein transforming the at least 
one digital speech value into coordinates comprises: 



-33- 



applying a mathematical function to the at least one 
digital speech value to produce a transform value, 
the range between the lowest possible transform 
value and the highest possible transform value 
being less than the range between the lowest 
possible digital speech value and the highest 
possible speech value; and 

using the transform value to identify coordinates for 
the at least one shape on the display. 

5. The method of claim 4 wherein applying a mathematical 
function comprises taking the logarithm of at least one digital 
value . 

6. The method of claim 4 wherein applying a mathematical 
function comprises taking the square-root of at least one digital 
value . 

7. The method of claim 4 wherein the shape is a base 
rectangle and wherein using the transform value to determine the 
coordinates of the base rectangle comprises: 

determining a base point for the base rectangle on the 
display; 

accessing a stored rectangle widths- 
accessing a maximum transform value; 

dividing the transform value by the maximum transform 

value to produce a transform ratio; 
determining a calculated height based in part on the 

transform ratio; and 
calculating the coordinates of the base rectangle based 

on the base point , the stored rectangle width and 

the calculated height. 



-34- 



8. The method of claim 7 wherein determining a calculated 
height comprises determining if the transform ratio is greater 
than a maximum height ratio for the base rectangle and if it is, 
performing a further step of multiplying the maximum height ratio 
for the base rectangle by a full meter height to produce the 
calculated height. 

9. The method of claim 8 further comprising steps of: 
subtracting the maximum height ratio for the base 

rectangle from the transform ratio to produce an 
excess ratio; 

determining a second rectangle height based in part on 
the excess ratio; and 

calculating the coordinates of a second rectangle based 
on the coordinates of the base rectangle, the 
stored rectangle width and the second rectangle 
height, the coordinates of the base rectangle and 
the second rectangle such that the second 
rectangle appears connected to a top edge of the 
base rectangle on the display. 

10. The method of claim 9 wherein determining a second 
rectangle height comprises determining if the excess ratio is 
greater than a maximum height ratio for the second rectangle and 
if it is, performing a further step of multiplying the maximum 
height ratio for the second rectangle by the full meter height to 
produce the second rectangle height. 



11. 



The method of claim 10 further comprising steps of: 
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subtracting the maximum height ratio for the' second 
rectangle from the excess ratio to produce a 
remainder ration- 
determining a third rectangle height by multiplying the 

remainder ratio by the full meter height; and 
calculating the coordinates of a third rectangle based 
on the coordinates of the second rectangle, the 
stored rectangle width and the third rectangle 
height, the coordinates of the second rectangle 
and the third rectangle such that the third 
rectangle appears connected to a top edge of the 
second rectangle on the display. 

12. The method of claim 7 further comprising calculating 
the coordinates of a background rectangle, the background 
rectangle appearing somewhere between the base rectangle and a 
point at a full meter height above a bottom edge of the base 
rectangle . 

13. The method of claim 1 wherein displaying a progress 
meter further comprises: 

dividing the speech input into frames; 

decoding at least one of the frames of speech into a 
sub-word unit; 

dividing a frame number of the last frame to be decoded 
by the total number of frames to produce a decode 
ratio; and 

displaying the progress meter based on the decode 
ratio. 



14. The method of claim 13 wherein displaying the progress 

meter further comprises: 
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multiplying the decode ratio by a full meter width to 

determine a progress width; and 
calculating the coordinates of a progress rectangle 

based on the progress width, a stored meter height 

and a base point on the display. 

15. The method of claim 12 further comprising: 
dividing the speech input into frames; 

decoding at least one of the frames of speech into a 
sub-word unit; 

dividing a frame number of the last frame to be decoded 
by the total number of frames to produce a decode 
ratio; and 

displaying the progress meter based on the decode ratio 
by changing the color of at least one background 
rectangle . 

16. The method of claim 15 wherein displaying the progress 
meter comprises : 

multiplying the decode ratio by a full meter width to 

produce a progress width; 
dividing the progress width by a rectangle width that 

is indicative of the width of each background 

rectangle, the division producing a rectangle 

count; and 

changing the color of a number of background 
rectangles, the number of background rectangles 
being equal to the rectangle count. 



17. 



A computer program comprising: 

at least one insertion point marker capable of 
maintaining the coordinates of an insertion point 



-37- 



on a display, the insertion point representing a 
location on the display where a user desires to 
provide input; 

a speech recognition routine capable of decoding a 
speech signal; and 

a meter generation routine capable of displaying a 
meter near the insertion point based on the 
insertion point marker, the meter being indicative 
of an amount of a speech signal that has been 
decoded by the speech recognition routine, 

18. The computer program of claim 17 wherein the meter 
generation routine further comprises: 

a microphone state variable having a value that is 
indicative of whether a microphone is active; and 

an active microphone display routine, capable of 
displaying an indication that the microphone is 
active near the insertion point. 

19. The computer program of claim 17 wherein the meter 
generation routine further comprises a transform routine capable 
of transforming a digital value into a set of coordinates for a 
shape on the display, the digital value being indicative of the 
magnitude of a portion of a speech signal. 

20. The computer program of claim 17 wherein the speech 

recognition routine is capable of decoding a 
speech signal into a set of sub-words. 

21. A method in a computer system for generating images on 
a display device, the method comprising: 
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receiving a speech input signal that is indicative of 
human speech; 

displaying a volume meter that is indicative of the 
magnitude of at least a portion of the speech 
input signal; and 

displaying a progress meter close to the volume meter 
on the display so that a user can perceive both 
the progress meter and the volume meter without 
substantially moving their eyes, the progress 
meter quantitatively indicating the amount of 
progress of a speech recognition system in 
decoding the speech input signal. 

22. The method of claim 21 wherein displaying a volume 

meter comprises: 

storing digital values representing the magnitudes of 
different respective portions of the speech 
signals- 
accessing the stored digital values; 

displaying a separate token for each separate digital 
value that is accessed. 



23. The method of claim 22 wherein displaying a separate 

token comprises: 

displaying a meter portion of the token, the meter 
portion's size being positively related to the 
magnitude of the speech signal such that higher 
magnitude portions of the speech signal have 
larger meter portions; and 
displaying a background portion of the token, the 
background portion's size being negatively related 
to the magnitude of the speech signal such that 
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higher magnitude portions of the speech signal 
have smaller background portions. 

24. The method of claim 23 wherein displaying the meter 
portion comprises : 

transforming a digital value representing the magnitude 
of a portion of the speech signal to produce a 
transform value, the range between the smallest 
and largest transform value being less than the 
range between the smallest and largest digital 
value; 

dividing the transform value by a maximum meter value 

to produce a meter ratio; and 
determining the height of at least a portion of the 

meter portion using the meter ratio and a full 

meter height. 

25. The method of claim 24 wherein determining the height 
of at least a portion of the meter portion comprises multiplying 
the meter ratio by the full meter height. 

26. The method of claim 24 wherein determining the height 
of at least a portion of the meter portion comprises: 

determining if the meter ratio is greater than a base 
ratio and if the meter ratio is greater than the 
base ratio performing steps comprising: 
multiplying the base ratio by the full meter 

height to determine the height of a base 

block of the meter portion; 
subtracting the base ratio from the meter ratio to 

produce an excess ratio; 
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using the excess ratio and the full meter height 
to determine a height of a second block of 
the meter portion. 

27. The method of claim 26 wherein using the excess ratio 
and the full meter height to determine a height of a second block 
comprises : 

determining if the excess ratio exceeds an intermediate 
ratio and if the excess ratio exceeds the 
intermediate ratio performing steps comprising: 
multiplying the intermediate ratio by the full 

meter height to produce the height of the 

second block; 
subtracting the intermediate ratio from the excess 

ratio to produce a remainder ratio; and 
multiplying the remainder ratio by the full meter 

height to produce a height for a top block of 

the meter portion. 

28. The method of claim 24 wherein the speech recognizer 
decodes the speech input signal by converting frames of the 
speech input signal into sub-words and wherein displaying a token 
comprises : 

dividing the number of the last frame decoded by the 
speech recognizer by a total number of frames that 
form the speech input signal to produce a progress 
ration- 
multiplying the progress ratio by a full meter width to 

produce a progress width; 
dividing the progress width by a token width to produce 
an affected number of tokens; and 
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for each of the affected number of tokens, setting the 
color of at least a portion of each token so that 
it is different from the color of other tokens. 



29. A computer program designed to operate in a computer 
system having a display , the computer program comprising: 

a volume meter portion capable of displaying a volume 
meter on the display that is indicative of the 
volume of a human speech signal; 

a speech recognition portion that is capable of 
converting the human speech signal into a set of 
sub-words; and 

a progress meter portion capable of displaying a 
progress meter on the display proximate the volume 
meter, the progress meter being quantitatively 
indicative of the amount of progress of the speech 
recognition portion in converting the human speech 
signal . 

30. The computer program of claim 2 9 wherein the volume 
meter portion comprises: 

meter size program code capable of determining a 
maximum dimension for the volume meter; 

volume ratio program code capable of calculating a 
volume ratio that is defined as a magnitude value 
associated with the human speech signal over a 
maximum possible magnitude value; and 

volume token program code capable of generating a 
volume token on the display that has a size that 
is determined from the volume ratio and the 
maximum dimension for the volume meter. 
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31. The computer program of claim 30 wherein the volume 
token program code comprises: 

positive relation program code capable of generating a 
foreground portion of the volume token that is 
larger for higher magnitude values associated with 
the human speech signal; and 

negative relation program code capable of generating a 
background portion of the volume token that is 
smaller for higher magnitude values associated 
with the human speech signal. 

32. The computer program of claim 31 wherein the progress 
meter portion comprises : 

progress ratio program code capable of dividing a frame 
number representing the last frame of the human 
speech signal converted by the speech recognition 
system by a total frame number representing the 
total number of frames found in the human speech 
signal to produce a progress ratio; 

meter dimension program code capable of determining a 
maximum dimension for. the progress meter; and 

progress dimension program code capable of multiplying 
the progress ratio by the maximum dimension for 
the progress. meter to produce a progress 
dimension . 

33.. The computer program of claim 32 wherein the volume 

token program code generates a volume token based in part on the 
progress dimension . 
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[57] ABSTRACT 

A speech recognition system provides a user with graphical 
and textual feedback. The textual feedback is displayed in 
windows but occupies little of the available display space 
and are displayed only for a short period of time. The 
graphical feedback is displayed in a designated notification 
area and does not obscure any other displayed . items. The 
feedback provided by the speech recognition system may 
indicate a current mode of operation of the speech recogni- 
tion system as well as a state of processing of audio input by 
the speech recognition system. 
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METHOD AND SYSTEM FOR DISPLAYING user. The computer system also includes a facility for 

SPEECH RECOGNITION STATUS displaying a graphical notification in the graphical notifica- 

INFORMATION IN A VISUAL tion area that identifies a current mode of operation of the 

NOTIFICATION AREA speech recognizer 

5 Id accordance with yet mother aspect of the present 

TECHNICAL FIELD invention a computer system includes a video display, an 

The present invention relates generally to computer sys- audio in Pf device "J* 4 speech recognizer. The computer 

terns and more particularly to speech recognition in com- system likewise, includes a facility for displaying textual 

outer systems feedback to the user regarding processing of the received 

10 audio input by the speech recognizer in a window that is 

BACKGROUND OF THE INVENTION onI y displayed for a predetermined period of time. 

In accordance with yet another aspect of the present 

Speech recognition systems provide facilities for recog- invention, a computer-readable storage medium is designed 

nizing components of speech in audio input. When a speaker f or use in a computer system that includes a video display 

speaks into an audio input device, the speech recognition 15 anc j an aumo device for receiving audio input from a 

systems process the audio input to recognize speech com- usen The computer-readable storage medium stores a speech 

ponents. One difficulty encountered with such speech rec- recognizer for recognizing components of speech in the 

ognition systems is the difficulty in informing the user of aum o input that is received by the audio input device. The 

how the audio input was interpreted and whether the audio computer-readable storage media also stores a first facility 

input was. properly received and processed. Certain conven- 20 for prov iding a graphical notification area on the video 

tional speech recognition systems have attempted to address display in which to display graphical notifications. The 

this problem by providing a user interface, such as a speech computer-readable storage medium further stores a second 

recognition window that provides feedback to the user about facility for displaying graphical notifications in the graphical 

the progress and results of speech recognition. notification area regarding the speech recognizer. 
Unfortunately, such windows often obscure substantial por- 25 

tions of the available display space on the video display. In BRIEF DESCRIPTION OF THE DRAWINGS 

addition, such windows are typically displayed as long as the A preferred embodiment of the present invention will be 

speech recognition system is running so that the windows described Mow with reference to the following drawir^. 

con tin 'ie to obscure portions of the video display while the « • . , , c 

, r t r J , M FIG. 1 is a block diagram of a computer system that is 

speech recognition system is running. 30 , c . . ,. c . 

r & suitable tor practicing the preferred embodiment of the 

SUMMARY OF THE INVENTION - present invention. 

FIG. 2 is an example illustrating the display of a state icon 

The present invention overcomes the limitations of the f or the voice facility in the visual notification area in 

prior art by displaying feedback to the user in a fashion that 35 accordance with the preferred embodiment of Lhe present 

minimizes the extent and duration to which other displayed invention. 

objects are obscured by the displayed feedback. In accor- FIG. 3 illustrates the display of a tool tip when a mouse 

dance with a first aspect of the present invention a method cursor ^ at a state icon for the voice faciUty in accor . 

is practiced in a computer system that has a video display dance ^ the preferred embodiment of the present inven- 

and an audio input device for receiving audio input. The ^ t j on 

computer system also includes a speech recognizer for plG 4A mus , rates ^ dj& q{ aQd 

recogmzmg components of speech n the aud.o input. The inforrDation in response to the changinf.he state of the voice 

speech recognizer operates in one of a number of different c ^ t , t r .. . . tt °^ ° 

r . * *• .u* *u ^ L - i facility to the not listening state. 

modes of operation. For this method, a graphical user 

interface that includes a designated notification area for A , . JIG. 4B illustrates the display of graphical and textual 

displaying graphical notifications is displayed on the video 45 ^rmatton »» response to the changing of the state of the 

display. The current mode of operation of the speech rec- vo,ce faciht y to the llstenm S state * 

ognizer is determined, and a graphical notification of the FIG - 4C illustrates the display of graphical and textual 

current mode of operation of the speech recognizer is information in response to the changing of the state of the 

displayed in the designated notification area. 5Q voicc facility to the standing-by state. 

In accordance with another aspect of the present FIG 5 is a flow chart illustrating the steps that are 

invention, audio input is received from the user through an performed to display an icon by the voice facility in the 

audio input device. The received audio input is processed by graphical notification area. 

a speech recognizer to attempt to recognize components of FIG, 6 is a flow chart illustrating the steps that are 

speech. Textual feedback regarding the processing of the 55 performed to remove an icon from the graphical notification 

received audio input is displayed in a window that is area. 

displayed for only a predetermined period of time, such as FIG, 7 illustrates a menu sequence that is displayed to 

a few seconds. enable a user to change current state of the voice facility per 

In accordance with a further embodiment of the present the preferred embodiment of the present invention, 

invention, the computer system includes a video display for 60 FIG. 8 illustrates the display of a tool tip that indicates that 

displaying video images and an audio input device for audio input was heard by the voice facility and the display 

receiving audio input from a user. The computer system of an icon that indicates that the voice facility is processing 

additionally includes a speech recognizer for recognizing the audio input. 

components of speech in audio input that is received by the FIG. 9 illustrates a tool tip that indicates that audio input 

audio input device. The computer system further includes a 65 was heard, not recognized .and too loud and also illustrates 

component for providing a graphical notification area on the an icon that indicates the audio input was not recognized by 

video display for displaying graphical notifications to the the voice facility. 
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FIG. 10 illustrates the display of a tool tip thai is generated states which the voice facility may assume. The voice 

when the signa] to noise ratio of received audio input is facility 28 may assumed listening state' wherein the voice 

below an acceptable level. facility listens for audio input and processes all audio input. 

FIG. 11 illustrates the display of a tool tip that is displayed The voice facility 28 may also assume a not listening state 

- w^r: voice facility receives no audio input signal when 5 wherein no audio input received by audio input device 24 is 

in the listening slate: processed. Lastly, the voice facility 28 may assume a 

FIG. 12 illustrates an example of the appeai^-cr «f the standing-by state wherein the voice facility analyses audio 

video display when a voice command is recognized by Uk- \n? Jlt that is received by the audio input device 24 but only 

voice facility. * s respuusiv* to selected voice commands, including voice 

lQ commands that are prefaced by a trigger phrase, such as 

DETAILED DESCRIPTION OF THE "computer," and voice commands, such as "start listening/* 

INVENTION that change the state of the voice facility. 

The preferred embodiment of the present invention pro- The current state of the voice facility 28 is reflected in an 

vides a method and system for providing speech recognition icon that is displayed within the taskbar visual notification 

status information to a user in a visual notification area. The area 34 (FIG. 2). FIG. 2 shows an example of the not 

information is displayed in a fashion that occupies very little listening state icon 36. The presence of the not listening icon 

of the available display space on the video display and much 36 1D thc taskbar vlsual notification area 34 indicates that the 

of the information is displayed in , transitorv or temporary v ° ice * "°f c ™ 1 ? ^mng t0 a ^to mput from 

fashion so that any obstruction that is caused by the dis- , audl ° m P* device , 24 *° ne ° f the stre^Dgths of displaying 
playing of the status information is only temporary. , n st f lcons in j he v * ual notlficatl ° n ™*3* * < hat ^provides 

i-t^ * 1- li c 20 a ™o wa an d easdy recognizable location for displaying 

FIG. 1 shows a block diagram of a computer system 10 such mformatioiL Moreover, the icons displayed within the 

that is suitable for practicing the preferred embodiment of visual not ifi cat j ori area 34 do not occupy any of the virtual 

the present invention. Although the computer system 10 of desktop surface 32 and, hence, do not obscure other dis- 

FIG. 1 includes only a single processor, those skilled in the played items. 

art will appreciate that the present invention may also be ^ If a user y^ts the mouse 22 (FIG. 1) or another input 

practiced in systems that employ multiple processors, device to position a mouse cursor 40 (FIG. 3) over a state 

including distributed systems. The computer system 10 of i corij a tool tip will be displayed. FIG. 3 shows an example 

FIG. 1 includes a central processing unit (CPU) 10 that is wherein a mouse cursor 40 is positioned to point at the not 

accessed to a primary memory 14 and secondary storage 16. listening si ate icon 36 which is displayed in the taskbar 

The secondary storage 16 mzy take many forms including a ^ visual notification area 34. A tool tip 38 is displayed to 

hard dirk drive. The computer system 10 also includes a indicate that the icon is associated with MICROSOFT 

keyboard 18, a video display 20, a mou^e 22 and an audio VOICE." The tool tip 38 is a small window that is dispiayer' 

input device 24. The audio input device 24 may take the in proximity to the icon and that contains text. The tool tip 

iVrm of a microphone that is coupled to a sound board. The is implemented as a clipped rectangle that displays text 

computer system 10 need not include all of these peripheral vs provided by the application associated with the icon. Thas, 

devices; rather, systems lacking one or more of these periph- the voice facility 28 provides the text that is to be displayed 

eral devices may be used to practice the present invention. within the clipped rectangle for the tool tip 38. The tool tip 

The memory 14 holds a copy of any operating system 26, 38 is only displayed for a predetermined period of time, such 

voice facility 28 and application programs 30, For illustra- as 2 or 3 seconds. After the expiration of the predetermined 

tive purposes in the discussion below, the operating system 40 time, the tool tip 38 is no longer displayed and, thus, no 

26 is assumed to be the "MICROSOFT' "WINDOWS" '95 longer occupies real estate on the virtual desktop 32. 

operating system sold by Microsoft Corporation of A separate icon is provided for each of the three states. 

Redmond, Wash. The voice facility 28 includes a speech FIG. 4B shows the listening state icon 44, and FIG. 4C 

recognition engine for recognizing speech components in shows the standing-by state icon 48. FlGS.,3,and 4Ashow t 

audio input and a voice command facility for responding to 45 the^non-listening state icon. When the voice facility state is 

voice commands that are recognized by the speech recog- changed, the state icon for the new state is displayed within 

nizer. The application programs 30 may call upon facilities the taskbar notification area 34. In addition, a tool tip that 

of the operating system 26 and the voice facility 28. identifies the new state is displayed. FIG. 4A shows the tool 

The preferred embodiment of the present invention relays tip 42 that is displayed when the state changes to the not' 

status information regarding the current state of the voice 50 listening state. FIG. 4B shows a tool tip 46 that is displayed 

facility 28. One way in which status information is relayed when the state is changed to the listening state, and FIG. 4C 

to user is by displaying icons within a visual notification shows the tool tip 50 that is displayed when the state is 

area. As is shown in FIG. 2, the operating system 26 has a changed to the standing-by state. The voice facility 28 is 

taskbar 33 in which buttons associated with active tasks are responsible for providing the text for such tool tips and 

displayed. /^redesignated portion of the taskbar 33 is 55 initiating display of the tool tips. 

designated as a taskbar visual notification area 34 in which_ FIG. 5 is a flow chart showing the steps that are performed 

application programs 30, the voice facility 28 and the in order for voice facility state icons to be displayed in the 

operating system 26 , may display icons to convey status taskbar visual notification area. Initially, the voice facility 28 

information to a user. The taskbar visual notification area is sends a message to the taskbar to display the appropriate 

described in more detail in co-pending application entitled 60 state icon (step 52). This message includes identification 

"System Provided Visual Notification Area", Ser. No. information that enables the taskbar to identify where the 

08/355,398, U.S. Pat. No. 5,617,526 which was filed on Dec. bitmap for the icon is stored. The taskbar then retrieves the 

13, 1994 and is assigned to a common assignee with the icon (step 54). Once the taskbar has retrieved the icon 

present application. The co-pending application is incorpo- bitmap, the taskbar displays the icon using thc bitmap (step 

rated by reference herein. 65 56). 

In order to understand status information that is displayed The removal of a state icon so that it no longer is 

by the voice facility 28, it is helpful to examine the different displayed in the taskbar visual notification area 34 is also 
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message- driven. First, the voice facility 28 sends a mess&g<>. bring the microphone closer" is displayed in a tool tip 76, 

to the taskbar to no longer display the state icon (step 58 in such as shown in FIG. 10. If no audio input signal at all is 

FIG. 6). The laskhar receives this message and, in. response, received, a tool tip 77 (FIG. 11) that displays the message "Is 

removes the icon from the taskbar visual notification area 34 your microphone plugged in?" is displayed, 
(step 60). If the voice facility is still active, the taskbar may 5 The preferred embodiment of the present invention also 

then proceed to display a new state icon in the taskbar visual provides feedback when the voice command input by a user 

notification area. is properly recognized. FIG. i2 shovs an example wherein 

The taskbar visual notification area 34 is a window. The the voice command "What can I sayV i> properly recog- 

taskbar knows where each icon is displayed within the nized. The recognition icon 78, which resembict, « check 
taskbar visual notification area 34. As such, when the mouse 10 mark, is displayed within the taskbar visual notification area 

cursor 50 is positioned over one of the state icons, the 34. In addition, a tool tip 80 is displayed with the phrase 

taskbar knows that the mouse cursor is pointing at one of the "What can I say?" The voice facility 28 invokes the script 

icons and asks the voice facility 28 what to do in response that is associated with the command. In the example shown 

to the mouse cursor pointing to the state icon. The voice in FIG. 12 a window 82 is opened that provides a list of 
facility 28 causes a tool tip to be displayed after the 15 global voice commands. Furthermore, the voice facility 28 

expiration of a predetermined time period if the mouse displays an additional tool tip 84 that explains operation of 

cursor remains pointing at the state icon the tool tip identi- the hierarchical list that is displayed in the window 82. 

tied that the state icon is associated with the "MICROSOFT While the present invention has been described with 

VOICE" application. reference to a preferred embodiment thereof, those skilled in 

As mentioned above, a user has the ability to select a 20 the art will appreciate that various changes in form and detail 

current state for the voice facility 28. One manner in which may be made without departing from the intended scope of 

the user may select the state of the voice facility 28 is the invention as defined in the appended claims. For 

through a state icon. If the user positions the mouse cursor example, tool tips that differ from those displayed in the 

50 so that it points'at the state icon that is displayed within figures may be used to practice the present invention, 
the taskbar visual notification area 34 and depresses one of 25 Similarly, different state icons may be used to practice the 

the mouse buttons a menu 62 (FIG. 7) is displayed. One of present invention, and different voice facility states (other 

the choices on the menu 62 is a "Set Listening Mode" menu than those described) may be used to practice the present 

option 64 that present a slide-out menu 66. Th^jsMe-oiit invention. 

meniL.66..displays, each .of t the ^available states l ,an£allows,a.. We claim: 

Ji±se- Jo select a. state by positiqmng,the.mouse.cursor,pn.one.. 3 ? 1. In a computer system uaving a viJeo display, an audio 

ojf the states and releasing the mouse button. The current input de-ice for receiving audio input from a user and a 

state is indicated by a check mark in the slide-out menu 66. speech recognizer for reccgniziog components of speech in 

Foe the example shown in FIG. 7, the voice facility 2S is audio input, said speech recognizer opera ling in a number of 

current:) : n the not listening state. Alternatively, a dialog different modes of operation, a method comprising the 

may be displayed rather than the slideout menu. 35 computer-implemented steps of: 

The voice facility 28 also provides the user with another providing a graphical user interface on the video display, 

type of status information. Specifically, the voice facility 28 said graphical user interface including a designated 

provides the user with feedback regarding the audio input - notification area for displaying graphical notifications; 

that has been entered by the user through the audio input ^ determining a first current mode of operation of the 

device 24. In response to audio input, tool tips are displayed, speech recognizer; 

and the state icon may be changed to icons that indicate the displaying a first graphical notification of the first current 

state of processing by the voice facility 28.. moc ie of operation of the speech recognizer in the 

FIG. 8 shows an example of what is displayed to a user designated notification area; 

immediately after the user enters audio input that was 45 determining a second current mode of operation of the 

sufficiently audible for the voice facility 28 to hear it. In speech recognizer; and 

particular, a tool tip 70 is displayed that indicates that the displaying a second graphical notification of the second 

audio input was "Heard." In addition, the state icon is current mode of the speech recognizer in the designated 

replaced with an icon 68 that indicates that the voice facility notification area, wherein the first notification differs 

28 is currently processing the audio input. If the voice 50 from the second graphical notification, 

facility does not recognize the voice commands that have 2 . The method of claim J wherein the first current mode 

been input as audio input by the user, the processing icon 68 0 f operation of the speech recognizer is determined to be a 

is replaced with a question mark icon 72 as shown in FIG. listening mode in which the speech recognizer is listening 

9. The question mark icon 72 is an intuitive indication that f or anc j processing audio input received at the audio input 

the voice facility 28 does not recognize the audio input. In 55 device. 

addition, the phrase "Not recognized" will be displayed 3. The method of claim 1 wherein the first current mode 

within a tool tip 74. 0 f operation of the speech recognizer is determined to be a 

One of the reasons that the audio input may not be non-listening mode in which the speech recognizer is not 

recognized is if the user is speaking too loudly into the audio listening for or processing the audio input received at the 

input device 24. The voice facility 28 determines that the 60 audio input device. 

audio input is too loud when the audio input exceeds a given 4. The method of claim 1 wherein the first current mode 
threshold of loudness. In such a case, the tool tip 74 also of operation of the speech recognizer is determined to be a 
includes the message "Please speak quieter." The audio standing-by mode in which the speech recognizer is listen- 
input may also be too quiet. The voice facility 28 conveys ing for and processing only selected audio input received at 
this feedback by displaying the message "Please speak 65 the audio input device. 

louder" in a tool tip. Similarly, when the signal to noise ratio 5. The method of claim 1 wherein the step of displaying 

of the received audio input is too low, the message "Please the graphical notification of the first current mode of opera- 
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tion of the speech recognizor id the designated noi^*»c?tion 
area comprises displaying an icon that identifies the tiir-t 
current mode of operation of the speech recognizer in the 
designated notification area. 

6. The method of claim 1 wherein the computer system 5 
rims an operating system and the operating system provides 
the graphical user interface. 

7. The method of claim 1 further comprising the step of 
displaying text in a separate window that identifies the 
second current mode of operation. 

8. The method of claim 7 further comprising the step of 
terminating the display of the text and the separate window 
after expiration of a predetermined time period. 

9. The method of claim 1 wherein the computer system 
further comprises a mouse with at least one button, said 
mouse for manipulating a mouse cursor that points to 
locations on the video display and wherein the method 
further comprises the step of displaying a user interface 
element that enables the user to change the first current mode 
of operation of the speech recognizer in response to the 
mouse cursor pointing at the graphical notification and the 
user clicking the button on the mouse, 

10. The method of claim 1 wherein the designated noti- 
fication area does not obscure any other video output on the 
video display. 

11. In a computer system having a video display, an audio 
input device for receiving audio input from a user and a 
speech recognizer for recognizing components of speech in 
audio input, a method comprising the computer- 
impiemented steps of: ^ 

receiving sudio input Crcm t\«; aacr inrougb the audio 
input device; 

processing the received audio inpu: with the speech 
i^cognizer to attempt to recognize components if 
speech in the received audio input; and ^ 

displaying textual feedback regarding the processing of 
the received audio input by the speech recognizer in a 
tool tip window that is only displayed for a predeter- 
mined period of time. 

12. The method of claim 11 wherein the displayed textual 40 
feedback indicates that the received audio input was recog- 
nized by the speech recognizer. 

13. The method of claim 12 wherein the displayed textual 
feedback indicates what the received audio input was rec- 
ognized as. 45 

14. The method of claim 11 wherein the displayed textual 
feedback indicates that the received audio input was heard 
by the speech recognizer. 

15. The method of claim 11 wherein the displayed textual 
feedback indicates that the received audio input was not 50 
recognized by the speech recognizer. 

16. The method of claim 11 wherein the displayed textual 
feedback indicates that there are problems with the received 
audio input that make it difficult for the speech recognizer to 
process the received audio input. 55 

17. The method of claim 11 further comprising the steps 

of: 

providing a designated notification area on the video 
display for displaying graphical notifications by appli- 
cation programs; 60 

displaying a graphical notification in the designated noti- 
fication area to provide feedback to the user about 
processing of the received audio input by the speech 
recognizer. 

18. The method of claim 17 wherein the displayed graphic 65 
notification indicates that the received audio input was 
recognized by the speech recognizer. 
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19. The method of claim 17 wherein the displayed graphi- 
cal notification indicates that the received audio input was 
not recognized by the speech recognizer. 

20. A enrjuter system comprising: 
a video display for displaying video images; 
an audio input device for receiving audio input from a 

user; 

a speech recognizer for recognizing cooipLMicnt* of 
speech in audio input received by the audio input 
device; 

a component for providing a graphical notification area on 
the video display for displaying different types of 
graphical notifications to the user; and 
a facility for displaying graphical notifications in the 
graphical notification area that identify current modes 
of operation of the speech recognizer. 

21. The computer system of claim 20 further comprising 
a facility for displaying text in a window that identifies the 
current mode of operation of the speech recognizer. 

22. The computer system of claim 21 wherein the facility 
for displaying text only displays in the window for a 
predetermined period of time. 

23. A computer system comprising: 
a video display for displaying video images; 
an audio input device for receiving audio input from a 

user; 

a speech recognizer for recognizing components of 
speech in audio input received by the audio input 
device; and - 
a first facility for displaying textual feedback to. the user 
regarding processing of the received audio input by 
speech recognizer in a tool tip winder' that is only , 
displayed for a predetermined period of time. 

24. The computer system of claim 23 further comprising: 
a second facility for providing a graphical notification 

area on the video display for displaying graphic noti- 
fications; 

a third facility for displaying a graphical notification in 
the graphical notification area that provides feedback 
regarding processing of the received audio input by the 
speech recognizer. 

25. In a computer system having a video display, an audio 
input device for receiving audio input from a user, a graphi- 
cal user interface on the video display, said graphical user 
interface including a designated notification area for dis- 
playing graphical notifications and a speech recognizer for 
recognizing components of speech in audio input, said 
speech recognizer operating in a number of different modes 
of operation, a computer-readable medium holding 
computer-executable instructions for performing a method 
comprising the computer-implemented steps of: 

determining a first current mode of operation of the 

speech recognizer; 
displaying a first graphical notification of the first current 
mode of operation of the speech recognizer in the 
designated notification area; 
determining a second current mode of operation of the 

speech recognizer; and 
displaying a second graphical notification of the second 
current mode of the speech recognizer in the designated 
notification area, wherein the first notification differs 
from the second graphical notification. 

26. The computer-readable medium of claim 25 wherein 
the first current mode of operation of the speech recognizer 
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is determined to be a TiSicnr*;^ mode in which the speech 
recognizer is listening for and pr-^ssing audio input 
received at the audio input device. 

27. The computer-readable medium of claim 25 wherein 
the first current mode of operation of the speech recognizer 
is determined to be a non -listening mode in which the speech 
recognizer is not listening for or processing the audio input 
received at the audio input device. 

28. The computer-readable medium of claim 25 wherein 
the first current mode of operation of the speech recognizer 
is determined to be a standing-by mode in which the speech 
recognizer is listening for and processing only selected 
audio input received at the audio input device. 

29. The computer-readable medium of claim 25 wherein 
the step of displaying the graphical notification of the first 
current mode of operation of the speech recognizer in the 
designated notification area comprises displaying an icon 
that identifies the first current mode of operation of the 
speech recognizer in the designated notification area. 

30. The computer-readable medium of claim 25 wherein 
the computer system runs an operating system and the 
operating system provides the graphical user interface. 

31. The computer-readable medium of claim 25 wherein 
the method further comprises the step of displaying text in 
a separate window that identifies the second current mode of 
operation. 

32. The computer-readable medium of claim 31 wherein 
the method further comprises the step of terminating the 
display of the text and the separate window after expiration 
of piedetermiiied tune period. 

33. Tne computer-readable medium of claim 25 whereiri 
lb designated notification area does not obscure any other 
video output on the video display. 
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34. In a computer system having a video display, an audio 
input device for receiving audio input from a user and a 
speech recognizer foi recognizing components of speech in 
audio input, a computer-readable medium holding 
computer-executable instructions for performing a method 
comprising the comp ,.L-i -implemented steps of: 

receiving audio input from the user through the audio 
input device; 

processing the received audio input with the speech 
recognizer to attempt to recognize components of 
speech in the received audio input; and 
displaying textual feedback regarding the processing of 
15 the received audio input by the speech recognizer in a 
tool tip window that is only displayed for a predeter- 
mined period of time. . 

35. The computer-readable medium of claim 34 wherein 
the displayed textual feedback indicates that the received 

20 audio input was recognized by the speech recognizer. 

36. The computer-readable medium of claim 35 wherein 
the displayed textual feedback indicates what the received 
audio input was recognized as. 

^ 37. The computer-readable medium of claim 34 wherein 
the displayed textual feedback indicates that the received 
audio input was not recognized by the speech recognizer. 

38. The computer-readable medium of claim 34 wherein 
the displayed textual feedback indicates that there are prob- 

30 lems with the received a:><dio ivvv* that make it difficult for 
the speech recojr-'.i^er ;o process the received audio input. 

***** 
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[57] ABSTRACT 

A multiple function graphical user interface for a speech 
recognition application adapted for generation by a com- 
puter programmed with a routine set of instructions, the 
interface comprising: an activa table icon defining an elon- 
gated screen display area, the icon having separately con- 
trollable foreground and background displays substantially 
coextensive with the display area; the background display 
being substantially fully changeable in response to user 
speech into a microphone having at least two states; text 
being displayable in the foreground display across substan- 
tially all of the display area; and, activation of the icon 
invoking a function related to the speech recognition appli- 
cation. A border can be selectively displayed around the 
elongated screen display area for indicating whether the 
speech application is in a navigation mode or a dictation 

31 Claims, 3 Drawing Sheets 
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MULTIPLE FUNCTION GRAPHICAL USER 
INTERFACE MINIBAR FOR SPEECH 
RECOGNITION 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates generally to the field of graphical 
user interfaces for speech applications, and in particular, to 
a multiple function graphical user interface for speech 
recognition which utilizes a minimum of screen space. 

2. Description of Related Art 

Presently, as developers add speech functions to computer 
graphical user interfaces, the speech function requires its 
own graphical interface. Speech users naturally want to 
minimize the amount of screen space occupied by the 
graphical user interface for speech recognition because the 
user can control the interface by voice. Generally speaking, 
the smaller the graphical user interface for speech 
recognition, the better. 

Integrating speech recognition into a visually-based 
. graphical user interface can be difficult. The difficulty stems 
from the fact that because the graphical user interface is 
visually-based, every encroachment upon this limited space 
reduces the free space for the user. However, the present 
level of speech recognition technology requires that the user 
be able to review several kinds of visual feedback. Therefore 
the challenge posed for developing an improved graphical 
speech interface Ls to present the minimal information 
required in the smallest space possible. 

Current technology requires that \tc us^r have available 
the following information: 1) the state of the microphone 
speech system, that is on, off or asleep; 2) the last recognized 
phrase; 3) whether ihc application has speech focus; 4) 
feedback that the application^ is vvorking;. and, 5) status 
messages from the speech system. In addition the user must 
also have, at a minimum, the ability to control the micro- 
phone state with the mouse or other pointing tool, in case 
speech is not functioning. At best the user must have a 
mouse activatable equivalent for every speech command, 
because speech is not a deterministic input system. 

Combining all of these functions into a truly minimal 
space is an elegant solution to a difficult problem. Various 
existing systems combine functionality in floating speech 
bars. Specific examples are instructive. 

Existing systems combine their functionality in floating 
speech bars. The Dragon systems is shown in FIGS. 12(a) 
and X2(by They combine the microphone and VU meter into 
one button, as shown in the upper right hand corner of FIG. 
12(b). 

In VOICE from Kurzweil, as shown in FIG. 13, all of the 
functions are combined on a tool bar and display the 
recognized text in a floating tool window. 

The tool bars from Voiceiype Connection and Voice 
Center, both available from IBM, are as shown in FIGS. 14 
and 15 respectively. 

None of the tool bar arrangements described above can 
truly be described as minimal, in terms of the necessary 
display area. 

SUMMARY OF THE INVENTION 

A minimal graphical user interface for speech recognition, 
also referred to herein as a minibar, in accordance with the 
inventive arrangements, combines the recognized text field, 
the on/off button, and the volume meter into one graphical 
.-user interface component. 
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A computer programmed with a routine set of instructions 
stored m a ohysical medium, for generating a multiple 
function grapui^ai user interface for a speech recognition 
application in accords*. ce with an inventive arrangement, 
5 comprises: means for display;:^ ?n activatable icon defining 
an elongated screen display area, the icor. having separately 
controllable foreground and background cuspid yr substan- 
tially coextensive with the display area; means for chau^' ^ 
substantially all of the background display in response to 
10 user speech into a microphone having at least two states; 
means for displaying text in the foreground display across 
substantially all of the display area; and, means responsive 
to activation of the icon for invoking a function related to the 
speech recognition application. 
15 The changing means changes can advantageously change 
the background display by one of the following: changing 
from a first color to at least a second color responsive to 
variations of the volume of the user speech; generating a 
color ribbon having one fixed edge and one movable edge 
20 responsive to variations of the volume of the user speech; 
and, changing between lighter and darker shades of at least 
one color responsive to variations of the volume of the user 
speech. 

The text displaying means, in conjunction with each 
25 embodiment of the changing means, can selectively display 
one or more of the following: each last recognized phrase of 
. the user speech; status messages from the speech applica- 
tion; and, prompts for changing the states of the microphone- 
In conjunction with each of the embodiments of the 
changing means, the function invoking means can display: 
a menu list of secondary speech functions responsive ro 
activation of a pointing tool button when the icon is dis- 
played; and, further icons for additional dictation functio -s 
35 can be displayed when dictation is active. 

In conjunction with each of the embodiments noted 
above, the computer advantageously further comprises 
means for selectively generating a border around the elon- 
gated screen display area and around the further icons for 
w indicating whether the speech application is in a navigation 
mode or a dictation mode. 

A multiple function graphical user interface for a speech 
recognition application adapted for generation by a com- 
puter programmed with a routine set of instructions, in 
45 accordance with another inventive arrangement, comprises: 
an activatable icon defining an elongated screen display 
area, the icon having separately controllable foreground and 
background displays substantially coextensive with the dis- 
»t,„ t,,jr™„„j K_; nn - ^,.u,-i — ♦;~ii„, a.h.. 

50 changeable in response to user speech into a microphone 
having at least two states; text being display able in the 
foreground display across substantially all of the display 
area; and, activation of the icon invoking a function related 
to the speech recognition application. 
55 The background display can be advantageously change- 
able responsive to variations of the volume of the user 
speech in at least one of: from a first color to at least a second 
color; as a color ribbon having at least one movable edge; 
and, between fighter and darker shades of at least one color. 
60 The text can advantageously includes at least one of; each 
last recognized phrase of the user speech; status messages 
from the speech application; and, prompts for changing the 
states of the microphone. 
The invoked functions can advantageously include at least 
65 one of: displaying a menu list of secondary speech func- 
tions; and, displaying further icons for additional dictation 
functions. 
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llie interface cao advantageously further comprise a - t-inul'y, being able to devote the entire length of the 

border selectively displayed around the elongated screen minibar to sx! also facilitates the display of status messages 

display area and around the further icons foi indicating from the speed* application. 

whether ih' -r^ch application is in a navigation mode or a A minimal graphical u>cr interface for speech recognition 

dictation mode. 5 combines the recognized text iiC: J, the on/off button, and the 

BRIEF DESCRIPTION OF THE DRAWINGS volum , e <™) ?T °™ " ^ T" 

ponent. Part ot the tnck in generating a muiup.o ninction 

FTGS. 1(a), 1(6) and 1(c) show a first embodiment of a display in a single component advantageously relies c:_ 

minimal graphical user interface tool bar in accordance with controlling the color or colors of the background of the 

the inventive arrangements. 10 component to represent the volume meter, which is difficult 

FIGS. 2(a) and 2(b) show a second embodiment of a to show in black and white drawings, 

minimal graphical user interface tool bar in accordance with FIG. 1(a) shows a minibar 10 which is arbitrarily cross 

the inventive arrangements. hatched to represent a light color 12, for example yellow. 

FIGS. 3(a) and 3(b) show multiple functionality based on FIG. 1(b) shows the minibar 10 arbitrarily cross hatched 

the first and second embodiments respectively. 15 with a denser pattern of lines to represent a darker color 14, 

.FIG. 4 shows a graphical user interface tool bar, as in for example green. FIG. 1(c) shows the minibar 10 arbi- 

eilher of the first and second embodiments, in a resting state traril y CTOSS hatched' with a still denser pattern of lines to 

and displaying the last command. represent a still darker color 16, for example red. The 

FIG. 5 shows a graphical user interface tool bar, as in 90 sequence of the yellow, green and red colors can be easily 

either of the first and second embodiments, in a resting state P^ rceived f presenting volume level of dictated speech, 

after a timeout function erases the last command shown in ^?™ e > * e pr °^ lon of hatching density in 

Pjq ^ FIGS. 1(a), 1(b) and 1(c) can also represent lighter and 

* ' , . .. . ■ r * i i_ ■ darker shades of the same color, which can also be easily 

• I J u 5 « WS 3 f aP . u ^ r ; nterface . too1 b , ar ' a * perceived as representing the volume level - of dictated 
either of the first and second embodiments, m a resting state 25 o nccc h 

when the microphone is asleep. A . t , „ , t 

-,t . An alternative background color display is shown in 

• ? & 7, Sh 2 WS 3 f ra P hlcal r "" erface too1 bar > 35 "> FIGS. 2(a) and 2(b), in which the minibar 10 is arbitrarily 
either of the first and second embodiments, in a resting state crQss hatdjed ^ a m q£ ^ tQ dis l i 

when the microphone is turner* ort. * . * 0 , • 

* volume as a moving nbbon or thermometer 18, wherein 

F'O ft shows a graphical user interface tool bar, as in 30 movement of the ri-ht edge 20 the ubbon, for exarrwic, 

either of the fust. and second embodiments, with an js easily percei ved as representing the volume level' of 

appended pu*. down menu. _ dicta-cd speech. 

FIG. 9 shows a graphical user interface tool bar, as in milli : w l0 can display the last dictated phrase ot 

cither of the first and seum J embodiments, in a navigation command 2 4, as shown in FIGS. 3(«) and 3(6), which dL&er 

mode and with additional function buttons. ~ from one anot her in the manner in which the background 

FLG. 10 shows a graphical user interface tool bar, as in co lor display is implemented, 
either of the first and second embodiments, in a dictation W hen no dictation has been received after a predeter- 

mode and with additional function buttons. ramed timeout or delay, the last phrase or command is 

FIG. 11 is a block diagram of a computer programmed m erased> v This condition can be displayed as shown in FIG. 5-' 

with a routine set of instructions for generating a multiple by using a symbol 28 such as a dashed line, 
function graphical user interface in accordance with the when the microphone is asleep, and must be awakened 

inventive arrangements, and as shown in FIGS. 1-10. for further dictation, the minibar 10 can appear as shown in 

FIGS. 12(a), 12(6) and 13-15 illustrate various speech FIG. 6. A status message 30 indicates the microphone is 

tool bar arrangements known in the prior art. 45 asleep and a prompt 32 is the proper command to wake up 

DETAILED DESCRIPTION OF THE the microphone. 

PREFERRED EMBODIMENTS When the microphone is turned off, and must be turned on 

for further dictation, the minibar 10 can appear as shown in 

The inventive arrangements embodied in a multiple func- HG 7 A status ~ lCS3agc 34 indicates the microphone is 

tion graphical user interface supply the required information 50 turned off and a prompt 36 is the proper commanc j t0 turn on 

in the smallest space possible. All of the following infor- tne microphone. 

mation can be provided. The minibar 10 can be controUe d, for example, by a 

The on, off or asleep state of the speech system is mouse. Clicking with the left mouse button on the minibar 

displayed. If the system is off, the preferred nature of the can tum tne minibar on and off. Clicking with the right 

display is text, which tends to be more understandable than 55 m0 use button can bring up a display of a list of secondary 

an icon. speech functions, for example, a menu 40 as shown in FIG. 

The entire length of the minibar, which can change during 8. When dictation is active in an application, additional 

the display and/or be user adjustable in length, can be dictation buttons 42 and 44, for example for starting and 

devoted to displaying the last recognized phrase or com- stopping dictation respectively, will become visible as 

mand. Having extra room for the text field is particularly 60 shown in FIGS. 9 and 10. The minibar in FIG. 9 is in a 

important for European, Middle Eastern and African lan- navigation mode, indicated by a border 46, preferably in a 

guages. contrasting color, around the basic minibar. Navigation 

Whether the application has speech focus is easily indi- commands, also referred to as control commands, will 

catcd with the large VU meter changing colors in the appear in place of the dashed line 28 as they are recognized, 

background, thus increasing the visibility of the active 65 The minibar ia FIG. 10 is in a dictation mode, indicated by 

speech application. A large volume meter changing colors the border 48 around the start and stop dictation buttons 42 

also alerts the user that the speech application is running. and 44. The border 46 is also shown in FIGS. 4, 5, 6 and 8, 
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as I he dictation application is in the navigation mode. The 
borders, which can also be shown in different colors to 
represent status conditions, provide yet another function in 
a minimal screen area. A double left-click can be used to 
invoke a fast-path function, for example launching an audio 
adjustment application. 

The minibar can be embedded into the Windows95 task 
bar; can be embedded into a tool bar in the application; can 
be embedded into the title bar of an application window; or, 
can be used as a floating window. 

A computer system 60 is shown in block diagram form in 
FIG. U. The computer system is programmed with a set of 
instructions stored in a physical medium, for example a hard 
drive 66 and/or a random access memory (RAM) 64 of a 
central processor 62, for generating the minibar 10 shown in 
FIGS. 1-10. The computer has an audio interface 80, 84 
between one or more sound or audio cards 76 in the 
computer apparatus and each of a microphone 78 and a 
speaker 82. Microphone 78 is intended to represent, for 
example, a headset microphone, a desktop microphone, a 
monitor microphone and a hand held microphone. Speaker 
82 is intended to represent, for example, one or more internal 
speakers, one or more external speakers, one or more 
monitor speakers or a headset speaker. The computer system 
60 has a graphics adapter 68 that generates the GUI displays 
explained above and shown in FIGS. 1-10. The computer 
system further comprises a monitor 70, a keyboard 72 and 
a mouse 74. The dashed line box shown in random access 
memory 64 represents a programmed routine of instructions 
for generating the graphical user interface minibar, in accoi- 
dance with the inventive ^.nangements. The institutions are 
stored in a physical medium embodied by hard drive 66 and 
loaded into another physical meJium embodied by random 
access memory 64. The programmed routine of instructions 
implements the steps of generating the minibar 10 as shown 
and described herein. 

This method of displaying the basic speech information to 
the user minimizes the required screen space. The interface 
taught herein provides information a user requires as long as 
the chance of speech misrecogaitions can occur, as long as 
there is a detectable delay in speech recognition, as long as 
the system has to provide the user with messages, and as 
long as the multiple windows paradigm is used on comput- 
ers. This will continue to be the environment of speech 
applications for the foreseeable future. 

What is claimed is: 

1. A computer programmed with a routine set of instruc- 
tions stored in a physical medium for generating a multiple 
function graphical user interface for a speech recognition 
application, comprising: 

means for displaying an acti vat able icon having a border 
defining an elongated screen display area within said 
border, said icon having separately controllable fore- 
ground and background displays substantially coexten- 
sive with said display area, said foreground display 
superimposed on said background display; 

means for changing substantially all of said background 
display in response to variations of the volume of user 
speech into a microphone having at least two states; 

means for displaying text in said foreground display 
across substantially all of said display area; and, 

means responsive to activation of said activatable icon for 
invoking a function related to said speech recognition 
application. 

2. TTie computer of claim 1, wherein said changing means 
changes said background display progressively from a first 
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color to * i least a second color responsive to variations of the 
volume ot user speech. 

3. The compute: cf claim 1, wherein said changing means 
changes said backgrous* 1 by generating a color ribbon 

5' having one fixed edge and one c ovablc edge responsive to 
variations of the volume of said user ^ j"-h. 

4. The computer of claim 1, wherein said chafing means 
changes said background display between lighter and ua.- V?r 
shades of at least one color responsive to variations of the 

10 volume of said user speech. 

5. The computer of claim 1, wherein said text displaying 
means displays each last recognized phrase of said user 
speech. 

6. The computer of claim 1, wherein said text displaying 
15 means displays status messages from said speech applica- 
tion. 

7. The computer of claim 1, wherein said text displaying 
means displays prompts for changing said states of said 
microphone. 

20 8. The computer of claim 1, wherein said text displaying 
means selectively displays: 

each last recognized phrase of said user speech; 

status messages from said speech application; and, 
2j prompts for changing said states of said microphone. 

9. The computer of claim 2, wherein said text displaying 
means selectively displays: 

each last recognized phrase of said user speech; 
status messages from said speech application; and, 
30 £.oinpts for changing said states of said microphone. 

10. The computer of claim 3, wherein said text displacing 
means t*t', actively displays: 

each last recognized phrase of said user speech; 
35 status messages from >aid spetch application; and, 
prompts for changing said states of said microphone. 

11. The computer of claim 4, wherein said text displaying 
means selectively displays: 

each last recognized phrase of said user speech; 
w status messages from said speech application; and, 
prompts for changing said states of said microphone. 

12. The computer of claim 1, wherein said invoked 
function displays a menu list of secondary speech functions. 

13. The computer of claim 12, wherein said invoked 
* 5 function displays further icons for additional dictation func- 
tions. 

14. The computer of claim 1, wherein said invoked 
function displays further icons for additional dictation func- 
tions. 

,0 15. The computer of claim 13, wherein said changing 
means changes said background display progressively from 
a first color to at least a second color responsive to variations 
of the volume of said user speech. 

16. The computer of claim 13, wherein said changing 
'* means changes said background by generating a color 

ribbon having one fixed edge and one movable edge respon- 
sive to variations of the volume of said user speech. 

17. The computer of claim 13, wherein said changing 
means changes said background display between lighter and 

0 darker shades of at least one color responsive to variations 
of the volume of said user speech. 

18. The computer of claim 13, wherein said text display- 
ing means selectively displays: 

each last recognized phrase of said user speech; 

status messages from said speech application; and, 

prompts for changing said states of said microphone. 
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19. The computer of claim 15. wherein said text display- separately controllable foreground and background dis- 
ing means selectively displays: plays substantially coextensive with said display area, 

each last recognized phrase of said user speech; said foreground display superimposed on said back- 
status messages from said speech application; and, ground display; 

prompts for changing said states of said microphone. said background display beinij, substantially fully change - 

20. The computer of claim 16, wherein said text display- " able in response to variations or volume of user 
ing means selectively displays: speech into a microphone having at least two states; 

each last recognized phrase of said user speech; text being displayable in said foreground display across 

status messages from said speech application; and, 10 substantially all of said display area; and, 

prompts for changing said states of said microphone. activation of said activatable icon invoking a function 

21. The computer of claim 17, wherein said text display- related to said speech recognition application. 

ing means selectively displays: 27. Tlie interface of claim 26, wherein said background 

each last recognized phrase of said user speech; display is changeable responsive to variations of the volume 

status messages from said speech application; and, 15 of aid user s P eech in at least oae of: 

prompts for changing said states of said microphone. progressively from a first color to at least a second color; 

22. The computer of claim 1, further comprising meaus as a color ribbon having at least one movable edge; and, 
for selectively generating a border around said elongated between lighter and darker shades of at least one color, 
screen display area for indicating whether said speech 20 28. The interface of claim 26, wherein said text includes 
application is in a navigation mode or a dictation mode. at j east Qne 0 f. 

23. The computer of claim 8, further comprising means , , , , c . , 

for selectively generating a border around said elongated each last ^cognized phrase of said user speech; 

screen display area for indicating whether said speech status messages from said speech application; and, 

application is in a navigation mode or a dictation mode. 25 prompts for changing said states of said microphone. 

24. The computer of claim 13, further comprising means~ 29. The interface of claim 26, wherein said invoked 
for selectively generating a border around said elongated functions include at least one of: 

screen display area and around said further icons for inch- , . t - , , t , 

, r . J ., .i--.. • displaying a menu list or secondary speech functions; and, 

eating whether said speech application is in a navigation 1 J ° J r 

mode or a dictation mr-de 30 displaying further icons for additional dictation. ^usclioiis 

25. The computer of claim *8, further comprising means 30 - ^ interface of claim 29, further competing, a tx>ru;u 
foi selectively generating < border around said elongated selectively displayed around said elongated screen display 
screen display area and around s?id further icons for indi- area and around said further icons for indicating whether 
cating whether said speech application is in a navigation said speech application is i n a navigation mode or a dictation 
mode or a dictation mode. 35 mode. 

26. A multiple function graphical user interface for a 31. The interface of claim 26, further comprising a border 
speech recognition application adapted for generation by a selectively displayed around said elongated screen display 
computer programmed with a routine set of instructions, area for indicating whether said speech application is in a 
said interface comprising: navigation mode or a dictation mode. 

an activatable icon having a border defining an elongated 40 

screen display area within said border, said icon having * + *■*/* 
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ABSTRACT 



In a voice actuated computer system voice command from 
an end user relevant to or promoted from a discrete location 
on a display screen are analyzed by the system. In response, 
a confirmation area is displayed on the screen at a location 
functionally related to the analyzed contents and context of 
the voice input or the screen location the utterance was 
prompted from. Within the confirmation area the computer 
interpretation of the utterance is displayed persisting and 
dissolving at selectively adjustable rates and times. Display 
of the recognized utterance is thereby placed in a confirma- 
tion area at variable locations where the user's focus is likely 
to be. Distractions are avoided associated with a fixed 
location confirmation area which obscures other content on 
the display screen and/or destroys end-user focus by requir- 
ing the eyes to shift from a location of current interest on the 
display screen to a different location wherein the confirma- 
tion is displayed. Persistence and dissolution time of the 
confirmation area and associated analyzed voice command , 
displayed therein may vary automatically as a function of 
the degree of confidence of the voice recognition system that 
the command was accurately analyzed, and correctly rec- 
ognized commands are automatically executed after display 
in the confirmation window. 

23 Claims, 3 Drawing Sheets 
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METHOD AND APPARATUS FOR looking for confirmation that the utterance he or she just 

PRESENTING PROXIMAL FEEDBACK IN made was in fact interpreted or acted upon correctly. 

VOICE COMMAND SYSTEMS Current speech recognition systems display this "best 

guess" in a control window at a fixed position on the screen 
CROSS REFERENCE TO RELATED 5 0 f the GUI in a control window. Although the spatial 

APPLICATIONS positioning of this control window may in some implemen- 

The present application is related to the following appli- t bc dtcrcd b ? as f ^ cd or / s " 
cations: (1) application Ser. No. 09/213,856, filed Dec. 17, * fixed locatloc ' such 35 ^ tltle b « of me cur ™ a PP^on 
1998,endded"S P eechCommandInputRecognitionS y stem 1fl bcm S .^^^^^^ 
for Interactive Computer Display With Interpretation of 10 ?*°c»ted .with ^technology. These methods of display- 
Ancillary Relevant Speech Query, Terms Into Commands" m & th H ese ^terpreted commands attempt to place the con- 
(IBM Docket AT9-98-343); (2) application Ser. No. 09/213, * * oslUon s ° *? to n °! b ock £ e V3Slblkt y 
858, filed Dec. 17, 1998, entitled "Speech Command Input ° f other ^? ortant P orUons of appUcation^ However, m 
Recognition System for Interactive Computer Display With „ dom S so > ^ ma y «c«itote distracting and disrupting eye 
Means for Concurrent and Modeless Distinguishing 15 movement from me area of me screen where the end-user is 
Between Speech Commands and Speech Queries for Local- fo u cuslD S to * e confirmation area of the voice recognition, 
ing Commands'* (IBM Docket AT9 -98-344); (3) application thereupon the usei -must then recal and reposition his or her 
Ser. No. 09/213,846, filed Dec. 17, 1998, entitled "Speech eyes at area of me screen which had the user s focus pnor 
Command Input Recognition System for Interactive Com- M t0 the rnterrupUon caused by display of the confirmaUon 
puter Display with Speech Controlled Display of Reoog- 20 ^ Moreover, current systems display the results of the 
nized Commands", (IBM Docket AT9-98-341); (4) apphca- s P eech P^ception system without taking into account inter- 
tion Ser. No. 09/213,845, filed Dec. 17, 1998, entitled P rctatlon of ^ command > 35 * » re P ortcd < 
"Speech Command Input Recognition System for Interac- It will be readily appreciated that m sessions of any length 
five Computer Controller Display with Speech Controller with such a voice recognition computer system these draw- 
Display of Recognized Commands" (IBM Docket AT9-98- backs can become extremely tiresome both physically and 
342). All of the above are assigned to a common assignee mentally and can severely impact the productivity of the 
and filed on the same day as the present patent application. end-user in a manner as to almost effectively negate the 

aforementioned benefits to voice recognition or navigation 

BACKGROUND OF THE INVENTION systems. Current methods and technology have simply been 

30 unable to eliminate these serious problems of maintaining 

1. Field of the Invention visibility of important areas of the display and the focusing 
This invention relates to voice command computer sys- and refocusing just described. 

terns and, more particularly, to such systems which present Accordingly, such a voice command and control system 

command feedback to the end-user. ^ was desired which could avoid the distractions to 

2. Background and Related Art end-users and the degradation of visibility of applications 
As the computer field has matured, vast improvements presented on GUI user interfaces. 

have been made in easing the manner in which end-users 

may interface with the systems. What was originally a crude SUMMARY OF THE INVENTION 

command fine interface requiring keyboard input of the 40 In a vo ice actuated computer system, voice input from an 

end-user has now evolved to a rich panoply of techniques end-user, such as command and/or control utterances rel- 

and devices for facilitating a more natural human interface evant t0 or prompted from a discrete location on a display 

with the computer. Representative examples of this include screen, is analyzed by the system. In a response to such 

various pointing devices such as mice, track balls, touch- analysis, a confirmation area is thereby displayed on the 

screens and voice activated interfaces made possible by 45 display screen in a location functionally related to the 

advances in computerized speech recognition. analyzed contents of the voice input or the location the 

Such speech recognition falls into two major categories. utterance was prompted from. Within the confirmation area 

First, systems have been developed for voice data input, the computer system's interpretation of the utterance, is 

examples of which include speech dictation or "speech-to- displayed for a preselected time and then gradually dis- 

text" systems such as those marketed by the IBM Corpora- 50 solved also at a preselected rate, such persistence and 

tion and Dragon Systems. Yet a second category of com- I dissolution being selectively adjustable by the end-user. In 

puterized speech recognition technology is represented by this manner, display of the interpretation of the command or 

command and control systems wherein end-user speech control utterance will be placed in a confirmation area which 

patterns are detected and analyzed. A specific form of this is m turn spatially positioned on the display screen at 

command and control technology is represented by a "desk- 55 variable locations functionally related to the content of the 

top navigator", wherein a user may navigate through a command or control utterance so as to be proximal to the 

computer system's graphical user interface (GUI) by merely location on the display screen which may have had the user's 

speaking menu command words, labels on buttons, direc- cye f ocus which gave rise to the utterance. In this 

tional commands such as "up" and "left" to move the manner, distractions are thereby avoided associated with a 

familiar mouse cursor, and the like. 6Q fi xc d location confirmation in the prior art (1) which 

Due to inherent voice recognition errors such speech obscures content of interest on the display screen and/or (2) 

recognition systems are prone to — partly because of the state destroys end-user focus by requiring the eyes to shift from 

of the technology and partly due to the variability of the* positions of interest on the display screen to a different 

user's speech patterns and memory of the correct inputs to location wherein the confirmation is displayed. An important 

utter, such systems typically reserve a dedicated, pre deter- 65 factor in the invention is the interpretation of the utterance 

mined area of the GUI to display the system's interpretation and therefore the true target of the command. Feedback will 

of the user's utterances. The user, by inspecting this area, is be preferably withheld, momentarily in most cases, until 
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such correct interpretation to application functionality can and refocus and to regain concentration on the content of this 

be determined. area 32. One reason historically for locating this confirma- 

In a more general case, the invention is not intended to be doD M6a 34 which typ«»Uy persists for a great deal of time 

limited to the prior art programmer's graphical model com- "? a out ° f the L wa X loc t ation f n me screen 30 f » 
prised of interaction and association with cursor targets, s mat docs not thereby obscure other relevant portions of the 

menu items, pointing devices, and toolbars. Rather, the s< * een 30 , whM * m >nformauonal content of the apph- 

7 t ° \ , . r j- i £ cation is to be displayed, 

invention contemplates a visual mechanism for display or ™ . -» * , , , , 

recognized speech commands in the form of natural lan- ^T 8 ? ? ™' ^P^. ." e , 

j i»* c ji_ i *j i * tL r correlative examples of screens 30 corresponding to that or 

guage and resulting feedback actions proximal to the area or . . . ™ r * L • ^ 

? • u- i • * c L i. * j 1 n the pnor art screen 30 of FIG. 1, wherein the features and 

focus in a graphical user interface which gave rise to and 10 , r Ci - x . \ . . . . , 

lj4 L r , jolt _n_ 1 benefits of the invention may be seen clearly depicted 

prompted the speech command. Such feedback actions may . „ c . „ , . ' * * • ™ 

• i j * * * * - i i v ^ j therem. Refemng first to FIG. 2 A, again a screen 30 on 

include text over text, visuals, color change, animation, and «■ . ■ j- 1 j r 

, , * c • tj- ■ / *i_ which is displayed a representative screen from a voice 

gradual reduction of an image or message fading mto the , . < \. . 

f . „ " , ■ j r> it i * *.l actuated application executing in the computer system of 

target area automatically so as to provide feedback to the A *F T t , • •* •« l * j tL . 

■ t iU **u / i . • * i . , . iL FIG. 4 is shown. In this screen 30 it will be noted that a 

end-user that the correct spoken action has been taken in the * s *• i * i i_ • i. t_- i_ t. 

, . , - . . r conventional tool bar 34 is shown which may have a 

desired focus or target area. „ , . t , tU . , . . J . 

pulldown menu 35 associated therewith in the upper left- 

BRIEF DESCRIPTION OF THE DRAWINGS hand comer of the screen 30. One voice recognizable 

command shown in this pulldown menu 35 might be "open 

FIG. 1 is a representative display screen in a voice 20 file" 37. When the end-user utters this "open file" command 
activated computer system demonstrating the method of the 37, as in the case of the interface of FIG. 1, the familiar 
prior art for displaying confirmation areas. confirmation area 36 may be made to appear proximal to the 

FIGS. 2A and 2B are illustrations of display screens in menu 35 wherein the voice-recognized utterance "open file" 
accordance with the invention for improved displays of W1 U appear (or whatever the system recognizes) thereby 
confirmation areas associated with voice command and 25 P rovi<iin g feedback to the user as to whether the user's 
control systems. speech was correctly recognized. 

FIG 3 is a flow diagram implementable in program code Similarly, turning to FIG. 2B, yet another pulldown menu 
executing in the computer system of the invention for 38 ma y a PP ear associated with the task or tool bar 34 at the 
effecting the improved confirmation area depicted in FIG. 2. u PP er P ort i°n of tne screen 30 associated with yet another 

i~rr A . fa ju.1 l j* j * < * -30 series of voice recognizable commands. It will be noted that 

FIG. 4 is a simplified block diagram depicting a voice JU „ , . & . A . iL „ rt ,.~ 

, . j . r . . * . i ,« c * c »i_ this pulldown menu is at a location on the screen 30 different 

actuated computer system in which the features of the - r 4 . , r ,™ <*a j *i_ * «j • 

j * i-i *. j from that of FIG. 2A, and that this pulldown menu might 

mvention are advantageously implemented. . t . ' 4 L1 y , , „ 

r have a different voice actuatable command such as print 

DETAILED DESCRIPTION ^ displayed in this pulldown menu. Upon the user uttering 

35 this voice actuatable command "print", in like manner to the 

Turning first to FIG. 1, depicted therein is a simplified case with respect to FIGS. 1 and 2 A, in this FIG. 2B will be 

illustration of a representative display 30 which might occur seen that a confirmation area 36 appears on the display 

on the monitor 39 of the system of FIG. 4 in accordance with screen 30 proximal to where the voice actuatable "print" 39 

the prior art. Id this system, a user might be interacting with command appears. 

and accordingly have eyes focused upon an area 32 of the 40 A comparison of these FIGS. 2 A and 2B with the prior art 

screen 30 which may prompt voice command to be analyzed user interface of FIG. 1 reveals an important and significant 

and acted upon in the system of FIG. 4. Once the voice difference. As previously described, in the case of the prior 

recognition system makes an attempt to recognize such an art FIG. 1 interface, the area of concentration of the end-user 

utterance, it has been conventional to display the computer's 32 may be spatially a significant distance from where the 

interpretation or translation of this command in a confirma- 45 default confirmation area 34 appears, giving rise to the 

tion area 34. For example, if the user is viewing an area 32 associated undesirable results of disrupting concentration, 

of an application in which there is menu of Voice activatable causing the necessity of eye refocus and the like as the eye 

choices such as "open dictionary", he or she may conse- traverses between areas 32 and 34. 

quently utter this command. In response thereto, the com- [ n contrast, however, it will be noted that in the user 

puier system, after recognizing the command or its best 50 interface examples of FIGS. 2A and 2B, a significant dif- 

guess as to an interpretation, this interpretation (such as fcrence is that this confirmation area 36 is displayed on the 

"open dictionary" if the utterance was correctly screen 30 in an area proximal to the area on the screen which 

so-interpreted) would appear in this confirmation area 34. gave t0 mc vo i ce command which in turn caused display 

A significant deficiency in this operation of voice com- of the confirmation area 36 and wherein the user's eyes are 

mand systems previously alluded to may be seen clearly 55 thus most likely to be focused. Thus, for example, in FIG. 

depicted with reference to this prior art screen 30 of FIG. 1. 2A, the confirmation area 36 is proximal, e.g., just below the 

It will be noted that whereas the end-user's focus was on location on screen 30 where the voice actuatable "open file" 

area 32, in response to a voice command prompted thereby command 35 is displayed in the pulldown menu which, 

he or she will receive feedback as to how the command has when uttered, gave rise to display of this confirmation area 

been interpreted in this confirmation area 34 which is 60 36 on the screen and the recognized "open file".. In like 

spatially separated a significant distance on the user inter- manner, in the illustration of FIG. 2B, this confirmation area 

face screen from the area 32 which prompted the command 36 now appears at a different location on the screen from that 

in the first place. This causes the user to have to refocus his of FIG. 2 A., namely at a location proximal to the "print" 

or her eyes at the diagonally opposite end of the screen 30 voice actuatable command 39 associated with pulldown 

at the confirmation area 34 in order to discern whether or not 65 menu 38 which itself appears in a different location from the 

the voice command was correctly interpreted. If so, the user pulldown menu 35 .including the "file" command of FIG. 

must then cause his or her eyes to traverse back to area 32 2A. In this manner, the user's eyes do not have to traverse 
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a disrupting significant distance from the location on the command shown at box 44 in response to viewing a cor- 
display screen which prompted the voice actuatable com- relative command displayed on the display 30. Such input 
mand in order to verify that the command was correctly may be made by\ means ^of a microphone 28 shown in the 

interpreted as viewed in the confirmation area 36. system of FIG. 4' and in response to reviewing one or more 

It is yet a further significant feature of the invention that 5 display screens 30 occurring on the monitor 39 of FIG. 4. 
in the course of the voice navigation program which ana- Once this voice recognition command has been uttered, 
lyzes and displays the interpreted command in the confir- the system of FIG. 4 will thereafter detect an appropriate 
mation area 36 that a 'context-sensitive analysis function spatial location for providing feedback to the end-user in a 
which may include elements of artificial intelligence as confirmation area 36, this step being illustrated by block 46 
desired and appropriate, may be built into the voice recog- 10 of FIG. 3. It will be recalled that this implementation of this 
nition system so as to position this confirmation area 36 functional block may include, as previously described, vari- 
proximal on the screen 30 to the intended and appropriate ovs factors as appropriate-including elements of artificial 
target of the command also displayed on the screen 30 which intelligence which have been sensing prior user interaction, 
gave rise to display of the confirmation area and the inter- ^ * e plication. Additionally such detection at block 46 
. j j j- 1 j ,u • tu * 1 may further include analysis of which voice actuatable; 
preted command displayed therein. Thus, for example, upon 15 J , t1 r . . , ,. . 
K , * • « £i » ^- l »• commands are currently being displayed on the display, 
the end-user uttering open file , this subroutine or program - A , , . ^> * • * «• * 
- ., . r . . ' • •_ screen 30, and may also incorporate intelligent predictions 
feature in the voice ^recognition program executing on the as to where ^ en V US er's ey £ ^ intellectual focus may 
system of FIG. 4 will determine that because this open file next QQCm Qn ^ ^ scfccn 3Q based of 
command is currently being displayed in the upper left-hand prior interac ti 0I1 with the voice actuatable program, 
corner of the screen 30 that the confirmation area 36 should 20 Continuing ^ nG> 3> oncc thc program co de has 
therefore appropriately be displayed somewhere proximal to dete cted the desirable spatial location for feedback in the 
where this "open file" command appears m the drop-down confirmation area 36, the system will thereafter cause dis- 
menu 35 of the toolbar 34. Similarly, upon the voice p i ay 0 f aQ appr opriate feedback message 48 in a confirma- 
recognition system detecting that the command "print" 39 tion area 36 located on the screen 30 (FIGS. 2A-2B) based 
has been uttered and correlating this to the fact that this 25 upon the analysis which previously transpired with reference 
command is displayed in the upper right comer of the to the function of box 46. It will be noted that one appro- 
display 30, the system of the invention will thereby deter- priate such feedback message might be simply displaying 
mine that, in contrast to the location of the display of the the alpha numeric version of the uttered command or 
confirmation area 36 of FIG. 2A, it would now be more instruction from the end-user as interpreted by the voice 
appropriate to display this confirmation area 36 in FIG. 2B 30 recognition program, e.g., displaying the words "open file" 
at a different location on the display screen, e.g., at a location in the confirmation area 36 upon detecting that it appears the 
proximal to where this "print" command 39 appeared which end-user has uttered the words "open file" in response to 
gave rise to the "print" utterance from the end-user. viewing this as a voice actuatable command choice in the 

Turning now to FIG. 3, depicted therein is a flow diagram pop-up menu 35. 

illustrating how program code would be provided executing 35 Once this "best guess" interpretation of the uttered voice 

on the system of FIG. 4 to implement the aforementioned command has thereby been displayed, 48, the process of 

features of the invention. First, it will be assumed that the FIG. 3 thereafter queries whether another message or com- 

end-user has invoked a voice recognition or navigation mand has been uttered by the end-user, shown by the 

application, as shown at box 40 which will be executing on decision block 50 of FIG. 3. If another message has been 

the system of FIG. 4. It may be desirable to the end-user to 40 detected, flow exits to the left of block 50 and is routed along 

adjust the persistence and dissipation of the confirmation path 52 back to block 44 wherein this next voice command 

area 36 prior to continuing with the process of FIG. 3. utterance input will be analyzed. Flow then continues ver- 

Accordingly, as shown by the dotted fines and the box 42, tically downward along the flow diagram of FIG. 3, where - 

this program code may be adapted to accommodate this upon this next desired spatial location for the confirmation 

feature. Alternatively, it will be appreciated that a feature of 45 area 36 for this next utterance is detected and the particular 

the invention may be to selectively alter this persistence and command analyzed. If, on the other hand, in response to the 

dissipation automatically, for example, as a function of the query at decision block 50, it has been determined that 

degree of confidence with which the voice recognition another voice command has not been entered or will not be 

system has recognized the utterance of the end-user. If, for entered (for example, because the voice navigable applica- 

example, the voice recognition system has almost a 100% 50 tion has been closed), the process exits to the right of 

confidence factor that the words "close application" have decision block 50 and the process ends, 54. 

been recognized, the persistence time and dissipation time of Before describing a block diagram of a computer system 

the confirmation message would desirably be lessened sub- in which the invention may be advantageously employed, a 

stantially in that it is less likely that this feedback would be few additional points must be noted. It should be readily 

of significance to the end-user. On the other hand, if the 55 apparent from the foregoing that once the confirmation area 

voice recognition application had difficulty in recognizing 36 has been extinguished after an appropriate length of 

an utterance, it would be desirable to automatically lengthen persistence and dissipation, if the command has been cor- 

the persistence and dissipation time of the confirmation rectly recognized it will automatically be executed by the 

message. This will allow the end-user more time to notice system. If, on the other hand, it has not been correctly 

that the command has not been interpreted correctly and to 60 recognized, while the confirmation area still persists and 

take corrective actions such as manually entering the correct before it has dissipated, the end-user may (depending upon 

desired command in the confirmation area or by manually a user-determined preference) override the recognized com- 

e fleeting the command through use of the keyboard or mand (or cause execution of a correct command in the event 

pointing device, or by invoking an undo feature. the command has not been recognized) by means of manual 

Continuing with FIG. 3, once the voice recognition appli- 65 entry in the keyboard, or use of a pointing device as 

cation program has been invoked and is executing, it will be required, or the utterance of a special voice command such 

assumed that the end-user will input a voice recognition as "STOP". 
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Yet an additional point is that although in the foregoing 
illustrations, voice recognition has been employed with 
respect to predetermined command choices for the end-user 
to select from a displayed pulldown menu or the like, the 
invention is not intended to be so limited. Accordingly, there 
may be voice actuatable actions not associated with mes- 
sages appearing on the screeo. Representative examples of 
this might be directional controls such as "move cursor 
down" or "enlarge figure". Such uttered directions may also, 
in like manner to a menu of command choices displayed on 
the screen, be recognizable by the voice system and acted 
upon, also with a confirmation of the words uttered appear- 
ing in a confirmation area 36 prior to execution of the 
recognized command or correction thereof to the keyboard 
or pointing device. Also, in keeping with a fundamental 
concept of the invention, even in such cases wherein a menu 
of spoken commands to select from is not present, it is 
contemplated that the voice recognition system will be 
command-context sensitive in the sense that it may intelli- 
gently determine where to place the confirmation area 36 
dependent upon the words uttered by the end -user. For 
example, if the end-user had uttered "move cursor to the 
right two inches", the system could determine by recogniz- 
ing the word "cursor" that some action associated with the 
cursor was desired — such as a move, shape change, or the 
like. Accordingly, the system would automatically display 
the confirmation area 36 with the recognized command 
displayed proximal to the current location of the cursor on 
the display screen 30. This, of course, is in recognition of the 
likelihood that the end-user will have focused his or her eyes 
immediately preceding the command on the cursor and 
thereby may be expecting a confirmation message to appear 
somewhere proximal to the current cursor location. 

FIG. 4 illustrates a preferred embodiment of a computer 
system which may advantageously employ the improved 
pointing device resolution system of the present invention. 
The system comprises a CPU 10, read only memory (ROM) 
11, random access memory (RAM) 12, I/O adapter 13, user 
interface adapter 18, communications adapter 14, and dis- 
play adapter 19, all interconnected via a common address/ 
data and control path or bus 16. Each of the above compo- 
nents accesses the common bus utilizing conventional 
techniques known to those of ordinary skill in the art, and 
includes such methods as dedicating particular address 
ranges to each component in the system, with the CPU being 
the busmaster. As is further shown in FIG. 4, these external 
devices such as DASD 15 interface to a common bus 16 
through respective adapters such as I/O adapter 13. Other 
external devices, such as the display 21, similarly use their 
respective adapter such as display adapter 19 to provide data 
flow between the bus 16 and the display 21 or other device. 
Various user interface means are provided for interconnec- 
tion and use with the user interface adapter 18, which, in the 
figure has attached thereto representative user input devices 
such as joy stick 23, mouse 25, keyboard 17, and speaker 
and/or microphone 27. Each of these units is well known in 
as such and accordingly will not be described herein. 

The invention admits to implementation on essentially 
any computer system and corresponding microprocessor, 
such as the RS/6000™, RISC-based workstations and per- 
sonal computers of the IBM Corporation executing the 
AIX™ and OS/2™ operating systems, respectively, or simi- 
lar machines of other vendors, which include for example in 
the case of an RS/6000 workstation a 604 PowerPC™ RISC 
chip. (RS/6000, IBM, AIX, OS/2 and PowerPC are trade- 
marks of the IBM Corporation). 

Contained with the CPU 10 of FIG. 4, typically is one or 
more microprocessors which performs the system address, 
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data, and control processing functions necessary for correct 
operation of the system of FIG. 4. Although the invention 
admits to application to various microprocessor designs, in 
the embodiment disclosed herein, the microprocessor takes 
the form of a PowerPC 604 microprocessor manufactured by 
the IBM Corporation, which is a species of microprocessor 
known as a reduced instruction set computer (RISC) micro- 
processor. Further details regarding the architecture and 
operation of such a microprocessor may be obtaiaed from 
the PowerPC 604 RISC Microprocessor Users Manual, 
Document #MPC604UM/AD 7 November, 1994, copyright 
IBM Corporation, which is incorporated herein by reference. 

In the context of the invention, the user will view various 
objects such as a cursor and pop up or pop down menus on 
the display 21 which may be manipulated by means of 
various pointing devices such as the mouse 25 and voice 
activated navigation. Program code associated with the user 
interface adapter i8 by way of a device driver for the 
pointing device 25 and microphone 27 in conjunction with 
operating environment and application code resident in 
RAM 12 and/or DASD 15 will facilitate and enable move- 
ment of a cursor on the display screen 21 responsive to and 
in association with correlative voice commands spoken into 
microphone 27. 

It will be understood from the foregoing description that 
various modifications and changes may be made in the 
preferred embodiment of the present invention without 
departing from its true spirit, It is intended that this descrip- 
tion is for purposes of illustration only and should not be 
construed in a limiting sense. The scope of this invention 
should be limited only by the language of the following 
claims. 

What is claimed is: 

1. A method for presenting feedback on a display in a 
voice command recognition computer system executing a 
voice recognition application, comprising: 

recognizing a speech command; 

selecting one of a plurality of positions on said display as 
a function of said recognized speech command; 

displaying feedback at a confirmation area proximal to 
said one of said positions and corresponding to said 
command; 

extinguishing said feedback at a time after said display- 
ing; • 

automatically executing said command after said display- 
ing; and 

wherein timing of said extinguishing is dynamically 
selected and wherein said feedback is selected from a 
group comprising a color change, animation, message, 
or reducing image on said display. 

2. The method of claim 1 further including 
determining a confidence factor corresponding to accu- 
racy of said recognizing; and 

wherein said time of extinguishing corresponds to said 
confidence factor. 

3. The method of claim 2 including 

depicting on said display information corresponding to 
said speech command at a location on said display; 

wherein said speech command corresponds to said infor- 
mation; and 

wherein said one of said positions corresponds to said 
location. 

4. The method of claim 2 including 
analyzing said speech command; and 

wherein said one of said positions is a function of said 
analyzing. 
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5. The method of claim 2 wherein said computer system 
includes 

a graphical user interface on said display and a pointing 
device; and 

wherein said speech command corresponds to a command 5 
displayed on said graphical user interface actuatable by 
said pointing device. 

6. The method of claim 2 wherein said confidence factor 
and said timing of extinguishing are inversely related. 

7. The method of claim 1 further including 10 
recognizing a next speech command; 

selecting a next one of a plurality of positions on said 
display screen as a function of said recognized next 
speech command; and 

displaying next feedback at a next confirm ation area 15 
proximal to said next one of said positions and corre- 
sponding to said next speech command, 

8. An apparatus for presenting feedback on a display in a 
voice command recognition computer system executing a 
voice recognition application, comprising: 

means for recognizing a speech command; 

means for selecting one of a plurality of positions on said 
display as a function of said recognized speech com- 
mand; 

means for displaying feedback at a confirmation area 
proximal to said one of said positions and correspond- 
ing to said command; 

means for extinguishing said feedback at a time after said 
displaying; means for automatically executing said 
command after said displaying; and 

wherein said timing of said extinguishing is dynamically 
selected and wherein said feedback is selected from a 
group comprising a color change, animation, message, 
or reducing image on said display. 

9. The apparatus of claim 8 further including 
means for determining a confidence factor corresponding 

to accuracy of said detecting; and 
wherein said time of extinguishing corresponds to said 
confidence factor. 

10. The apparatus of claim 9 including 

means for depicting on said display information corre- 
sponding to said speech command at a location on said 
display; 

wherein said speech command corresponds to said infor- 
mation; and 

wherein said one of said positions corresponds to said 
location. 

11. The apparatus of claim 9 including 
means for analyzing said speech command; and 
wherein said one of said positions is a function of said 

analyzing, 

12. The apparatus of claim 9 wherein said computer 
system includes 

a graphical user interface on said display and a pointing 
device; and 

wherein said speech command corresponds to a command 
displayed on said graphical user interface actuatable by 
said pointing device. 60 

13. The apparatus of claim 9 wherein said computer 
system includes 

a graphical user interface on said. display and a pointing 
device; and 

wherein said speech command corresponds to a non- 65 
visible command interpreted by an application running 
on said graphical environment. 
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14. The apparatus of claim 9 wherein said confidence 
factor and said timing of extinguishing are inversely related; 
and wherein said computer system further includes 

a graphical user interface on said display and a pointing 
device; and 

wherein said speech command corresponds to a non- 
visible command interpreted by an application execut- 
ing in said graphical user interface. 

15. The apparatus of claim 8 further including 
means for recognizing a next speech command; 
means for selecting a next one of a plurality of positions 

on said display screen as a function of said recognized 
next speech command; and 
means for displaying next feedback at a next confirmation 
area proximal to said next one of said positions and 
corresponding to said next speech command. 

16. A program product for presenting feedback on a 
display in a voice command recognition computer system 
executing a voice recognition application, comprising: 

program code means for recognizing a speech command; 

program code means for selecting one of a plurality of 
positions on said display as a function of said recog- 
nized speech command; and 

program code means for displaying feedback at a confir- 
mation area proximal to said one of said positions and 
corresponding to said command; 

program code means for extinguishing said feedback at a 
time after said displaying; 

program code means for automatically executing said 
command after said displaying; and 

wherein said timing of said extinguishing is dynamically 
selected and wherein said feedback is selected from a 
group comprising a color change, animation, message, 
or reducing image on said display. 

17. The program product of claim 16 further including 
program code means for determining a confidence factor 

corresponding to accuracy of said recognizing a speech 
command; and 
wherein said timing of extinguishing corresponds to said 
confidence factor. 

18. The program product of claim 17 including 
program code means for depicting on said display infor- 
. mation corresponding to said speech command at a 

location on said display; 
wherein said speech command corresponds to said infor- 
mation; and 

wherein said one of said positions corresponds to said 
location. 

19. The program product of claim 17 including 
program code means for analyzing said speech command; 

and 

wherein said one of said positions is a function of said 
analyzing. 

20. The program product of claim 17 wherein said com- 
puter system includes 

a graphical user interface on said display and a pointing 
device; and 

wherein said speech command corresponds to a command 
displayed on said graphical user interface actuatable by 
said pointing device. 

21. The program product of claim 17 wherein said com- 
puter system includes 

a graphical user interface on said display and a pointing 
device; and 



US 6,233 

11 

wherein said speech command corresponds to a non- 
visible command interpreted by an application running 
on said graphical environment 

22. The program product of claim 17 wherein said con- 
fidence factor and said timing of extinguishing are inversely 5 
related. 

23. The program product of claim 16 further including 
program code means for recognizing a next speech com- . 

mand; 
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program code means for selecting a next one of a plurality 
of positions on said display screen as a function of said 
recognized next speech command; and 

program code means for displaying next feedback at a 
next confirmation area proximal to said next one of said 
positions and corresponding to said next speech com- 
mand. 

* * * » * 
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ABSTRACT 



A system and method for management of an interface for 
communications systems and devices using a recognizer 
having a limited time duration recognition window, based on 
mapping available time to a spatial representation. The 
system has particular application to management of a speech 
interface for a speech recognizer with a limited time dura- 
tion recognition window. When the speech recognizer is on 
and a recognition window is opened, a timing signal is sent 
to the device and processed to provide spatial representation 
of the time remaining, e.g. in the form of an animation on a 
graphical user interface, or a recognizable pattern of stimu- 
lation using a haptic interface. As the recognition window 
advances i and closes, the animation or pattern also changes 
and closes, e.g. diminishes in size, to provide the user with 
spatial information indicative of the time left in the recog- 
nition window. A reset feature allows users to reopen the 
recognition window to delete and correct input, while the 
window is still open, or after closing the window, and before 
the system advances to an erroneous state. The system is 
particularly applicable to speech recognizers having a lim- 
ited recognition window that access speech recognition 
engines resident on a network or at a terminal device. 

29 Claims, 4 Drawing Sheets 
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COMMUNICATION SYSTEM USER accents, the user speaking to softly, may adversely affect 

INTERFACE WITH ANIMATED recognition performance. 

REPRESENTATION OF TIME REMAINING When conditions are not optimum, additional dialogue 

FOR INPUT TO RECOGNIZER may assist. For example, the recognizer may give repeat 

5 instructions, or provide additional instructions. 

RELATED APPLICATIONS Nevertheless, using speech to provide additional informa- 

Tbii application is related to U.S. patent application Ser. tion * dow - Consequently the user may perceive an exces- 

No. 09/062,969 entitled "Server for handling multimodal sivel y lon g wait for the svstem t0 reset and ^ a aew 

information" to H. Pasternak; and U.S. patent application P 10 ^ 1 - Typically, speech is perceived as fast for input, and 

Ser. No. 09/062,970 entitled 'Management of speech and 30 slow for out P ut - 

audio prompts in multimodal interfaces' to M. French St. Many us ers re P ort becoming frustrated with using inter- 
George, filed concurrently herewith. active voice response (IVR) systems offering many choices 

or a multi level menu system of choices. The user may forget 

FIELD OF THE INVENTION long lists of instructions, or become confused or lost in a 

„ . . . . • . - . . r 15 complex speech application. 

This mvention relates to management of interfaces for • .«-,... . ... 

telecommunications systems and devices having a limited User difficulties in mteracting with these systems repre- 
time window for capturing input, with particular application sent * ome r ^ asons f ch s ^ cch ^jfaces have not yet 
to speech recognizers having a limited time window for ^ n&d as ******* acceptance as they might, 
responding to a speech prompt. 20 01der svstems whlch also provide a graphical user 

interface, i.e. a screen display, with a speech interface, have 
BACKGROUND OF THE INVENTION been discrete non-integrated techniques That is the system 

W1 . ' . ' . , , . . may use either a touch input or a speech input, but not both 

Telecommunications systems with a speech recognition. simultaneously 

capability have been in use for some time for performing „, " . . - . . . « _ 

, . 4 * , • . a* • rr« 1 r « c To overcome the inconvenience of switching between 

basis tasks such as directory dialling. There are also network 25 . . & 

basedspeech recognition servers that deliver speech enabled dls f ete a PP^ tl0 °f «*mng dirferen modes of interacaon, 
j • ■ r j • 11 « °~ . , , tl * 11 i_ systems are being developed to handle more than one type 

directory dialling to any telephone. Typically, speech i . , , . - # . r , c . , 

* j • * ^- u l. • *- °f interface, l.e^cu^tharijjne^mode, of . input, and output, 

prompted interlaces have been used in telecommunications ^ . r f1 , ^ — ^~~-*-*?-~r~ rM *r ~ — ~ r ..r-.~>~ 

# - . ♦ u .u • 1 1 *u ^sxmultaneously^In the followuig description the term input/ 

systems in contexts where there is no visual display or the ^ '-7 rv.; " c . . i-. 1 : 

J « , , 1 1 * * c r i on output modality refers to a sensory modality relating to a 

user is unable to use visual displays, for example, a con- 30 , . . *••.*_• . t , 4 . ' . . . 

i*iu *, - i user s behaviour ui mteracnng with the system, i.e. by using 

ventional telephone terminal. . A -i j • 1 r .7*1 * r 

r auditory, tactile and visual senses. Input/output modes refer 

Topically, Jhe.spee^ to t0 specific examples of use of these modalities. For example 

§ES!*L!?3L JPi^Y 1 ! 1 ?? a si55^..EESffiEl»^^..l££98B^fe speech and audio input/output represent an auditory modal- 
Pte^I^uc^ it y; use of a keypad, pen, and touch sensitive buttons 

aJspeak_nowJ,^beep^fter which Jhe_speech..recognizer~is... repreS ent a tactile input modality, and viewing a graphical 
temedLan, Jor^a_Umited Jime ...window,., typically^a -few display relies on the mo daIity. 
seconds, oWg which the user may respond. t ^ example of a multimodal interface is 08/992,630 

Users of telecommunications equipment employing entitled "Multimodal User Interface", filed Dec. 19, 1997, to 
speech recognition systems often report feeling rushed when ^ Smith and Beaton, which is incorporated herein by refer- 
prompted to respond immediately after a beep or other e nee. Tins application discloses a multi-modal user interface 
audible prompt. anc j provides^! telecommunications system and methods to 

Part of this rushed feeling may be attributed to a sense that facilitate multiple modes of interfacing with users for 
" the device will stop recognition before the user has com- example, using voice, hard keys, touch sensitive soft key 
pleted their verbal request because the user receives no 45 input, and pen input. This system provides, e.g. for voice or 
indication of the time window available to respond after the key input of data, and for graphical and speech data output, 
recognizer it turned on or when the recognition window is The user may choose to use the most convenient mode of 
open. The user may find it difficult to know when the speech interaction with the system and the system responds to input 
recognizer is on, may talk when the recognizer is off, or may . from all modes. 

.become confused by no response. .50 Thus, interfaces for communications devices and corn- 

Other difficulties may occur if the user does not remember puter systems are becoming increasingly able to accept input 
what is acceptable input vocabulary or grammar to use. In and provide output by various modes, 
addition to the sense of having to respond right now, current For example, current speech recognition interfaces may 
speech interface structures do not provide the user with an be used in association with an visual display showing an 
opportunity to rephrase a request, or change their mind 55 icon that indicates current word recognition state. These 
before waiting for the system to respond. The user's utter- icons change visually when the recognition state changes 
ance is accepted and interpreted, and the system advances to fr om listening to not-listening. For example, a "talk now" 
the next logical state, which may result in an error, for icon ma y be displayed in the corner of the screen. While 
example if the user mis-speaks, coughs, or simply makes a these icons indicate to the user that the speech recognizer is 
mistake. Similarly, undue hesitation after a partial response, ^ 0 n, the icons do not overcome the users perception of 
may cause the system to move prematurely to the next urgency to talk before the window closes. Also, as men- 
logical state. If this state is not where the user wants to be, tioned above if an error is made, or speech input is inter- 
the user must navigate back to the previous state and restate rupted by extraneous background noise, the system waits 
the request. until the 'talk now' or recognition window closes, and 

Currently, the best recognizers in use have a 90 to 95 65 advances to the next logical state to recover from such an 
percent recognition performance under optimum conditions, error, before issuing a new prompt and reopening the rec- 
and a noisy background environment, other speakers, user ognition window. 
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There also exist natural language speech interfaces that 
are always on, which preclude the need for beeps that inform 
the user of when to start talking. The user may speak at any 
time, and the recognizer will always be ready to listen. 
Currently this type of recognition is not yet widely distrib- 5 
uted and used. These more advanced speech recognizers 
currently rely on a network based speech recognizer to 
provide the necessary processing power. Thus in the fore- 
seeable future, this type of advanced speech recognition will 
co-exist with simpler forms of recognition that require a 
limited duration 'time to talk* window, or recognition win- 
dow. 

SUMMARY OF THE INVENTION 

Thus, the present invention seeks to provide a system and 
method for management of an interface for a communica- 15 
tions systems or devices including a limited time duration 
window for accepting input, and particularly for manage- 
ment of a speech interface for a speech recognizer with a 
limited duration 'time-to-talk' window, which avoids or 
reduces some of the above mentioned problems. 20 

Therefore, according to a first aspect of the present 
invention there is provided a communications system com- 
prising: 

an interface operable for receiving input during limited 
time recognition window, 25 

means for mapping available time of the recognition 
window to a spatial representation in animated form using 
one of a graphical modality, haptic modality or auditory 
modality. 

For example the spatial representation comprises one of a 30 
two dimensional representation and three dimensional rep- 
resentation which diminishes in size as the available time 
diminishes, thereby providing a user with an indication of 
the available time to provide input. 

Where the means for mapping comprises a graphical user 35 
interface, preferably the spatial representation comprises 
graphical information in the form of an animation, which 
diminishes in size as the available time diminishes. Ihexate 
of size change o Ttheju^ 

lisl^b^ired, tolnfluence. the users perception of the remain- 40 
'ingTime ay ailable. . 

Alternatively, the means for mapping comprises a haptic 
interface and the spatial representation comprises an ani- 
mated pattern of haptic stimulation, e.g. produced by a 
wristband stimulator. " 45 

Alternatively, the means for mapping comprises an audi- 
tory interface the spatial representation comprises a time 
dependent auditory pattern. 

These alternative modalities may be combined, for 5Q 
example to provide both graphical and haptic feedback to 
the user regarding available time in the recognition window. 

Preferably, the system also provide a means for resetting 
the recognition window by the user, and reinitiating the 
animation. For example, the reset means may be a key or 55 
button associated with the animation. Alternatively, a touch 
sensitive region may be associated with the animation, e.g a 
touch sensitive button associated with the animation. 

According to another aspect of the invention there is 
provided a communications device comprising: 5Q 

an speech interface for accessing a speech recognizer 
operable for receiving speech input during limited time 
recognition window, and 

means for mapping available time of the recognition 
window to a spatial representation in animated form using 65 
one of a graphical modality, haptic modality or auditory 
modality. 



,711 

4 

According to yet another aspect of the invention there is 
provided a communications system comprising: 

a speech interface for accessing a speech recognizer 
operable for receiving speech input during limited time 
recognition window, 

means for mapping available time of the recognition 
window to a spatial representation in animated form using 
one of a graphical modality, haptic modality or auditory 
modality. 

Thus where the interface is a speech interface and the 
animation is an animation displayed on the graphical 
interface, the animation itself may provide the touch sensi- 
tive region which shrinks as the recognition window closes, 
and which is operable by the user to reset the 'time to talk' 
recognition window. An appropriate graphical animation for 
a speech interface is a speech balloon. 

Preferably, the system comprises means for resetting the 
recognition window, which comprises a tactile input means 
associated with the animation displayed on the graphical 
user interface, e.g. a button. 

For example when the animation is provided by a touch, 
sensitive region of the graphical user interface, the means for 
resetting the recognition window comprises, an associated 
touch sensitive, region of the graphical user interface. 

Where speech input has already been captured, the reset 
means also functions to delete speech input when the speech 
recognition window is reset. 

According to another aspect of the present invention there 
is provided a method of providing user feedback and control 
for a communications device for accessing a recognizer for 
receiving input while the recognizer is operable during a 
limited time duration recognition, comprising the steps of: 

after prompting the user for input; 

turning on the recognizer for a limited time duration 
recognition window; 

and while the recognizer is on, mapping available time to 
a spatial representation in animated form in one of a graphi- 
cal modality, haptic modality and auditory modality. . 

Where the interface comprises a graphical user interface 
the method comprises providing a spatial representation in 
the form of an graphical animation indicative of the time 
remaining in the recognition window, and preferably com- 
prises displaying an animation which diminishes in size as 
the time remaining in the recognition window decreases. 

This method is particularly useful for communications 
device having a speech interface for accessing a speech 
recognizer, where the speech recognizer is operable for 
receiving input during a limited time duration recognition 
window. 

Beneficially while the speech recognizer is on, the method 
comprises generating a timing signal representative of the 
time remaining in the recognition window, 

controlling the animation using the timing signal and 

at each time increment of the timing signal capturing 
speech input, and checking for user input to a reset means, 

and when user input to the reset means is captured, 
resetting the recognition window, deleting captured speech 
input, and reinitiating the timing signal and the animation, 

and when the timing signal corresponds to the end of the 
recognition window, capturing aggregated speech input and 
closing the recognition window. 

Another aspect of the present invention provides software 
on a computer readable medium for carrying out these 
methods. 
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Conveniently, in use of a speech interface, the graphical 
information is an animation displayed on the graphical user 
interface which changes as the recognition window closes 
with time. For example the animation may be a speech 
balloon, or other appropriate icon, which diminishes in size 5 
as the recognition window closes. The animation provides 
the user with a graphical representation of the time remain- 
ing to talk. Movement on the screen also has the advantage 
of drawing the users attention to the closing recognition 
window. 10 

Thus the user receives a visual indication of the time 
remaining in the recognition window of the speech 
recognizer, and is provided with the option to reset the 
recognition window, for example, to correct a speech input 
error. Thus the user receives feedback regarding the opera- 15 
tion of the speech recognizer, and is provided with an 
opportunity to control opening of the recognition window, 
e.g. to correct errors. 

Beneficially, by using a multitasking, multimodal user 
interface, and combining a touch sensitive display to the. 20 
interface, an animated graphic can also serve as a reset 
button that the user may touch, to re-open a closing window. 

This user behaviour resets the recognition window and 
allows the user to correct or alter a spoken command to a 
speech interface without having to wait for the system to 25 
achieve its next state before returning to correct an input. 
Reset means allows the user reset the time window at will. 
If not reset the animation stops when the time reaches the 
time duration Tw and the recognition window closes, after 
capturing aggregated speech input collected while the win- 
dow is open. 

Provision of an animated graphical output, with or with- 
out, haptic and auditory feedback, indicative of time 
available, provides the user with additional timing informa- 35 
tion. Consequently user feedback and control of the inter- 
face or speech interface is improved. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will now be described in greater detail with 
reference to the attached drawings wherein: 40 

FIG. 1 shows a schematic block diagram of a communi- 
cations network comprising a mobile telephone having a 
multitasking graphical user interface consistent with an 
embodiment of the present invention; 

FIG. 2 shows schematically a diagram of mobile tele- 
phone of a first embodiment for operation in the network of 
FIG. 1; 

FIG. 3 shows part of the mobile telephone of FIG. 2 
showing on an enlarged scale detail of the touch sensitive SQ 
graphical display with pen input capability, during operation ' 
of a Yellow Pages directory application using a method 
according to an embodiment of the invention. 

FIGS. 4A, 4B, 4C, and 4D show schematically part of the 
display shown in FIG. 3, showing on an enlarged scale, to 55 
show the time to talk indicator and controller displayed by 
the graphical interface as the time window of the speech 
recognizer advances; 

FIG. 5 shows schematically part of a graphical user 
interface representing an second embodiment of the indica- 50 
tor and controller; 

FIG. 6 shows schematically part of a graphical user 
interface representing an third embodiment of the indicator 
and controller; 

FIG. 7 shows schematically part of a graphical user 65 
interface representing an fourth embodiment of the indicator 
and controller; 
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FIG. 8 shows a flow chart setting out the steps of a method 
of providing interface management according to an embodi- 
ment of the present invention, i.e. user feedback and control 
of a speech interface using a multimodal interface of the 
mobile telephone shown in FIGS. 2 and 3. 

DETAILED DESCRIPTION OF THE 
INVENTION 

A schematic block diagram of a communications network 
10 is shown in FIG. 1 and represents a GSM switching 
services fabric 20 and a network services provider 40 
associated with a plurality of communications terminals, for 
example a mobile telephone 100, and other wired or wireless 
communications devices and terminals represented sche- 
matically by units 110, 120, 130, 140 and 150. 

The wireless mobile phone 100 according to a first 
embodiment of the present invention is shown enlarged in 
FIG. 2, and is provided with a multitasking graphical user 
interface, similar to that described in copending U.S. appli- 
cation Ser. No. entitled "Multimodal User Interface" filed 
Dec. 19, 1997, 08/992,630 to Smith and Beaton, which is 
incorporated herein by reference. This is a multi-modal user 
interface which provides a telecommunications system and 
methods to facilitate multiple modes of interfacing with 
users for example, using voice, hard keys, touch sensitive 
soft key input, and pen input. This system provides, e.g. for 
voice or key input of data, and for graphical and speech data 
output. Thus the user may choose to use the most convenient 
mode of interaction with the system and the system responds 
to input from all modes. 

As shown in FIG. 2, mobile telephone unit 100 comprises 
a body 200 which carries a display screen 210 for the 
graphical user interface, which may include touch sensitive 
buttons ; 212; a conventional keypad 220 and other hard keys 
222; speaker 240 associated with the speech interface to 
providing speech prompts as shown schematically by 230 to 
illustrate the various modes of interaction which may be 
selected by a user. Thus the system provides for conven- 
tional key input, a graphical user interface which includes 
touch sensitive soft keys, and a speech interface for output- 
ting speech and audio prompts, and a speech recognizer to 
accept and interpret speech input. Optionally, the graphical 
user interface may support pen input to allow written input 
on a touch sensitive area of the screen. 

The device 100 according to the first embodiment of the 
present invention comprises means for mapping available 
time of the recognition window to a spatial representation in 
one of a graphical, haptic or auditory modality. This embodi- 
ment is an example using a graphical modality for providing 
user feedback and control of a 'time to talk' recognition 
window of the speech interface. In FIG. 2, and in the 
enlarged schematic of the screen display shown in FIG. 3, 
near the bottom of the display screen 210, the user feedback 
means is an indicator which takes the form of an appropriate 
animated icon, or animation 216, as will be described below, 
and is associated with a touch sensitive button 218. The 
button 218 functions to control and reset the speech recog- 
nizer. The lower part of the screen 210 is shown enlarged in 
FIGS. 4A, 4B, 4C and 4D to show how the animation 
changes, i.e. shrinks, as the time to talk window of the 
speech recognizer is opened and closes. 

Since speech and word recognition systems have limited 
recognition windows, when the speech recogoizer is turned 
on and a recognition window is opened, a timing signal is 
sent to the device that starts the indicator, for example 
displaying the animation shown on a graphical display 
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represented in FIGS. 4A to 4D. The signal contains infor- A method in accordance with the first embodiment of the 

rnatioa regarding the timing of the animation, and as the present invention, with reference to the portable wireless 

recognition window advances and closes, the animation also terminal 100 shown in FIGS. 2 to 4, will be described in 

changes and closes. In this example the animation is a more detail with reference to the flow chart shown in FIG. 

speech balloon which diminishes in size as the time to talk 5 8. 

window closes. The timing signal contains information In operation of the device, the use may pick up the device, 

regarding the timing of the animation. As the recognition and cause activation of the device either automatically upon 

window closes , the animation also closes. picking up the device, or for example by pressing a button. 

The rate at which the animation diminishes in size is ^ speech reC ognizer is thereby switched on, and 

controlled by the timing signal. The rate of change of the 10 prompts ^ ^ for mput (A) ^ prompt may be m 

animation may be linear or non-linear with respect to the either by or m <earcon ^ that ^ a rec ognizable sound 

time available. While the user does not receive a count such „ a characteristic < beep ». ^ speech re cognizer (SR) 

down, i.e. it is not a numeric display of the time remaining, wind(W & mmQ6 OQ ^ f()r a ed&c reC ognition window 

the ammation provides a spatial representation of the time T . T ^ md a timing signal ^ mitiated at wfaich fe 

remaining, which allows the user to gauge how much time 15 mdicative of ^ time remaining, Tw, in the recognition 

is left. The user receives sufficient visual feedback on timing window 

information to provide and indication of the progress of the a..-^^ . j , ~ ^. 

window, and warning when the recognition window will * T/£ ' J" a j» re ? t « d 1 * ce ? h m ? nt ^l 6 S f° * 

close. The rate of change of the animation may be used to "P^ rcd <9 "f a l?™;"^. animation associated with 

influence the users perception of the available time and the 20 timing s^nal ^ delayed on the screen and the anima- 

rate at which the window is closing. . hon » started < D >- ^ ^ ech b ( a ! loon B ^ 

T iL . , . ° - . A ± An , speech recognition window, or time to talk window, is 

In the example shown in FIGS. 2, 3 and 4A to 4D, the / , * r\ • . . ^ > A ' 

. . . r . . , „ r„ L . * i_ displayed at its maximum size as shown m FIG. 4A. 

animation is a graphical speech balloon. Touching the button , , . 

218 associated with the animation, i.e. on an touch sensitive M cach s P« clfi ^ time interval or time increment, t, an 

adjacent region of the screen display, sends a signal to the 25 aggregated speech input sample is collected S^+S,. After 

recognizer to delete any verbal signal associated with a each s P eech m P ut 15 sam P led > a check * made to determine 

previously opened recognition window in this state, and * the s ? ccch recognizer window is open, i.e. if T^Tw. 

resets the timing signal, and the animation, indicating that When the time value is checked, if the speech recognizer 

the user may restate their request. recognition window has closed at T-Tw the aggregated 

Thus, the user is provided with information regarding the 30 s&&h sample is sent for speech recognition (G). 

time left in the recognition window, and an undo or reset If the time window is still open T<Tw, the recognizer 

feature is also provided. The reset feature allows users to checks whether the user has issued a reset command, i.e. 

correct speech utterances while the window is still open, or pressed the reset button, to reset the recognition window 

after closing the window, and before the system advances to (H). If not, the timing signal advances, the collection of the 

an erroneous state. 35 next input speech sample is commenced (E). The animation 

Thus the speech interface is provided with animated continues to advance, diminishing in size continuously as 

graphical support, to indicate the time duration of an avail- . tQe timing signal advances (see FIGS. 4B to 4D), to provide 

able recognition window. the user with a further, visual indication of the diminishing 

Optionally, haptic support is provided in addition to available time to talk left in the recognition window, 

graphical output, for example by providing a spatial repre- 40 When T=Tw, the aggregated speech sample is captured, 

sentation of the time remaining through a haptic pattern the recognition is closed, i.e. the recognizer is turned off, and 

generated by an array of stimulators carried on an associated the animation ceases. 

wristband of the communications device. If at any time during the recognition window, the user 

While in this first embodiment, the balloon animation and 45 resets the speech recognizer, any aggregated speech input 

balloon button are separate entities, advantageously, accord- that has been captured is erased, and a prompt for new 

ing to a second embodiment, the animation is provided on a speech input is issued, e.g. using a simple 'earcon'. The 

touch sensitive area of the display, i.e. the animation itself is timing signal is reset, (I) and the ammation is reset to the 

touch sensitive and functions as a soft button providing the time zero display as shown in FIG. 4A. 

reset function, as shown schematically in FIG. 5. 5Q The method of operation of the indicator and controller is 

Alternatively a soft key implementation of the indicator, generally applicable to all devices that access speech rec- 

i.e. the latter balloon button could alternatively is by a hard ognition engines, whether these are implemented locally, or 

key button version, as shown schematically in FIG. 6. The on a network server. 

animation illustrated above shrinks in length as the available The system and method described above was developed 

time diminishes, but the animation may alternatively take 55 particularly to facilitate interaction with a speech recognizer 
any convenient form, and inanother embodiment shown^in- The system and method in general is not limited to speech 

FIG. 7, the animation shrinks in 2 dimensions. These are interfaces, but may also be applied to other interfaces 

simply given as examples, and any convenient form of operating with a limited time window for receiving input, 

animation may be used. Another example may be a system for receiving and recog- 

The user is provided with graphical and dynamic infor- nizing security information e.g. user id and password 

mation regarding the time left in the recognition window. information, to provide access to an information system, 

The reset, or undo feature allows users to correct speech wherein a time limit for access is provided, for example to 

utterances before the system advances to an erroneous state. deter unauthorized entry. 

Speech recognition may reside on the network, or at the In another example (not shown) the haptic animation 

terminal device, and the indicator and controller is appli- 65 mentioned 1 above may be provided instead of graphical 

cable to speech and word recognition systems having limited information regarding the available time in the recognition 

recognition windows. window. In another example (not shown), the animation is 
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auditory, i.e. a pattern of sound representative of the avail- means for mapping available time of the recognition 

able time may be generated by stereo speakers of the window to a spatial representation in animated form 

communications device. using one of a graphical modality, hap tic modality or 

While the systems and methods described above may be auditory modality, 

implemented by any convenient software and hardware 5 12. A system according to claim 11 wherein the means for 

configuration for local device or network server implemen- mapping comprises a graphical user interface and the spatial 

tation of a speech interface, and the method is intended to be representation comprises a graphical animation, 

platform independent. For Internet compatible communica- 13 A communications system according to claim 12 

tions devices the current application is a preferred imple- comprising means for resetting the recognition window, 

mentation of the current application uses a Java web server, 10 14 A 00MunicaliDlls syste m according to claim 13 

such as one designed for this applicaUon as described in the ^ m eans for resetting the recognition window 

above referenced copending U.S. patent application Ser. No. . ♦ • * j -*u u. 

09/062,969, entitled "Server for handling multimodal infer- f 0m P tlSe f a ™* "P* ™™ wtn the 

mation" to H. Pasternak filed concurrently herewith, and ta* **fky* °° the pphid user interface, 

incorporated herein by reference. This server was developed u 15 ' A communications system according to claim 13 

for handling information in different modal forms associated 15 wherein the animation is provided by a touch sensitive 

with respective input/output modalities of a multimodal K & oa of ' he g™P^al user interface, and the means for 

interface, and may be used to implement the systems and rese " m S the "cognition window comprises an associated 

methods described herein. ,ouch sensitive region of the graphical user mterface. 

Although specific embodiments of the invention have 16 A t s ^ taa accordin S t0 cl f ID } i ^. herein me , s P a,ial 

u j «7 j ■ j * *i ** *it i_ * * , .,, j . 20 representation comprises one of a two dimensional repre- 

been described in detail, it will be apparent to one skilled in *\ .. . .. K. . t . .. . . 

^.u* • *• a ac J * *t. u a- * sentation and three dimensional representation which dimm- 

the art that vanations and modifications to the embodiments . , . . - .. tl 4 . *\. . . « 

. , .... % '■■ ishes m size as the available time diminishes, 

may be made within the scope of the following claims. . j- * i • -m l • *t. c 

\\tu i * a • 17. A system according to claun 11 wherein the means for 

What is claimed is: . , . . 4 _ , iL 

i . .. . . . mapping comprises a haptic mterface and the spatial rep re - 

1. A commumcations system comprising: . .. • • . , ^ . • , 

. x _ _ . . . . . . , .25 sentation comprises an animated pattern of haptic stimula- 

an mterface operable for receiving input during limited ^ on 

time recognition window, ^ Asystem according to claim 11 wherein the animation 

means for mapping available time of the recognition diminishes in size as the available time diminishes, 

window to a spatial representation in animated form 19. Asystem according to claim 18 wherein the rate of 

using one of a graphical modality, haptic modality or x change m size of the animaaon fe fo^. 

auditory modality 20. A system according to claim 18 wherein the rate of 

2. A system according to claim 1 wherein the spatial change m size of animation is non -linear. 
representation comprises one of a two dimensional repre- 2 1. Asystem according to claim 11 wherein the animation 
sentation and three dimensional representation which dimin- ^ mdicative 0 f foe operation of the recognition window, and 
ishes in size as the available time diminishes. 35 the Nation changes during the operation of the recogni- 

3. A system according to claim 1 wherein the means for lion window tQ indicate the remaioing time dura tion of the 
mappmg comprises a graphical user interface 1 and the spatial recognition window 

representation comprises graphical information in the form 2 2. A method of providing user feedback and control for 

of an animation. a commutations device for accessing a recognizer for 

4. A system according to claim 3 wherein the animation ^ receiving mput wniie me recognizer is operable during a 
diminishes in size as the available time diminishes. limited time duratkm KCOgnii i ont comprising the steps of: 

5. A system according to claim 4 wherein the rate of - . . - . 
change of the animation* linear. after P rom P tin g the *™ f ° r *P«* 

6. A system according to claim 4 wherein the rate of turning on the recognizer for a limited time duration 
change of animation is non-linear. 45 recognition window; 

7. A system according to claim 1 wherein the means for and while the speech recognizer is on, mapping available 
mapping comprises a haptic interface and the spatial repre- time, to a spatial representation in animated form in one 
sentation comprises an animated pattern of haptic stimula- of a graphical modality, haptic modality and auditory 
tion. modality. 

8. A system according to claim 1 wherein the means for 50 23. A method according to claim 22 wherein the interface 
mapping comprises an auditory interface the spatial repre- comprises a graphical user interface and the method com- 
sentation comprises an time dependent auditory pattern. prises providing a spatial representation in the form of an 

9. Asystem according to claim 1 comprising reset means graphical animation indicative of the time remaining in the 
for resetting the recognition window and reinitiating the recognition window. 

animation. 55 24. A method according to claim 23 comprising display- 

10. A communications device comprising: ing an animation which diminishes in size as the time 
an speech interface for accessing a speech recognizer remaining in the recognition window decreases. 

operable for receiving speech input during limited time 25 ■ A method of providing user feedback and control for 

recognition window, and a communications device having a speech interface for 

means for mapping Available time of the recognition 60 accessirj g a s P* Qch recognizer, the speech recognizer being 

window to a spatial representation in animated form operable for receiving input during a limited time duration 

usiag one of a graphical modality, haptic modality or recognition window, comprising the steps of: 

auditory modality. after prompting the user for input; 

11. A communications system comprising: turning on the recognizer for a limited time duration 
a speech interface for accessing a speech recognizer 65 . recognition window; 

operable for receiving speech input during limited time and while the speech recognizer is on, mapping available 

recognition window, time to a spatial representation in animated form using 
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one of a graphical modality, haptic modality and audi- 
tory modality. 

26. A method according to claim 25 wherein the interface 
comprises a graphical user interface and the method com- 
prises providing a spatial representation in the form of an 5 
graphical animation indicative of the time remaining in the 
recognition window. 

27. A method according to claim 26 comprising display- 
ing an animation which diminishes in size as the time 
remaining in the recognition window decreases. 

28. A method according to claim 27 comprising, 10 
while the speech recognizer is on, generating a timin g 

signal representative of the time remaining in the 

recognition window, 
controlling the animation using the timing signal and 
at each time increment of the timing signal capturing 15 

speech input, and checking for user input to a reset 

means, 

and when user input to the reset means is captured, 
resetting the recognition window, deleting captured 
speech input, and reinitiating the timing signal and the 
animation, 



and when the timing signal corresponds to the end of the 
recognition window, capturing aggregated speech input 
and closing the recognition window. 
29. Software on a computer readable medium for carrying 
out a method of providing user feedback and control for a 
communications device having a speech interface for 
accessing a speech recognizer, the speech recognizer being 
operable for receiving input during a limited time duration 
recognition window, comprising the steps of: 

after prompting the user for input; 

turning on the recognizer for a limited time duration 
recognition window; 

and while the speech recognizer is on, mapping available 
time to a spatial representation in animated form using 
one of a graphical modality, haptic modality and audi- 
tory modality. 
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signs was not legally erroneous, and because 
we find that the Board's finding that Valu's 
guide rails are dejure functional is supported 
by substantial evidence, the Board's refusal to 
register Valu's guide rail designs is affirmed, 
and Rexnord's cross-appeal is dismissed as 
moot. • - ■ ■ ; 

. . - AFFIRMED 



COSTS 



No costs. 
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PATENTS 

[1] Practice and procedure in Patent and 
Trademark Office — Board of Patent 
Appeals and Interferences — In gen- 
eral (§110.1101) ' • 

PatentabUity/Validity — Obviousness- 
Combining references (§ il5.09.05)';'": 

Patentability/Validity — Obviousness — 
Evidence of (§ 115.0906) 

Rejection of patent application for obvious-' 
ness under 35 U.S.C. §103 must be based on 
evidence comprehended by language of that 
section, and search for and analysis of prior 
art includes evidence relevant to finding of 
whether there is teaching, motivation, or sug- 
gestion to select and combine references re- 
lied on as evidence of obviousness; factual in- 
quiry whether to combine references must be 
thorough and searching, based on objective 
evidence of record, and Board of Patent Ap- 
peals and Interferences must explain reasons 
why one of ordinary skill in art would have 
been motivated to select references and to 
combine them to render claimed invention ob- 
vious. 



[2] Patentability/Validity — Obviousness 
— Combining references (§ 115.0905) 

JUDICIAL PRACTICE AND 
PROCEDURE 

Procedure — Judicial review — Stan- 
dard of review —r Patents 
(§410.4607.09) 

Board of Patent Appeals and Interferences 
improperly relied upon "common knowledge 
and common sense" of person of ordinary 
skill in art to find invention of patent applica- 
tion obvious over combination of two prior art 
references, since factual question of motiva- 
tion to select and combine references is mate- 
rial to patentability, and could not be resolved 
on subjective belief and unknown authority, 

. since deferential review .of agency decisions 
under Administrative Procedure Act reinforces 
obligation of board to develop evidentiary ba- 
sis for its findings, since board's rejection of 

•need for any specific hint or suggestion in par- 
ticular reference to support combination con- 
stituted omission of relevant factor required 

. by precedent, and thus was both legal error 
and arbitrary agency, action, since board's 
findings must extend to all material facts and 
be documented on record, and since "com- 
mon knowledge and common sense" are not 
specialized knowledge and expertise, of 
agency contemplated by APA, and may not be 
substituted for evidence,', although they may' 
be applied; to analysis of evidence. 

PATENTS 

[3] Practice and' procedure in Patent and 
Trademark Office — Board of Patent 
Appeals and Interferences — In gen- 
eral (§ 110.1101) 

Patentability/Validity — Obviousness — 
Evidence of (§ 115.0906) 

JUDICIAL PRACTICE AND 
-J PROCEDURE 

Procedure — Judicial review — Stan- 
dard of review — Patents 
(§ 410.4607.09) 

Patent examiners and Board of Patent Ap- 
peals and Interferences, in relying on what 
they assert to be general knowledge to negate 
patentability on ground of obviousness, must 
articulate that knowledge and place it on 
record, since examiners and board are pre- 
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sumed to act from viewpoint of person of or- 
dinary skill in art in finding relevant facts, as- 
sessing significance of prior art, and making 
ultimate determination, of obviousness issue; 
failure to do so is not consistent with either ef- 
fective administrative procedure or effective 
judicial review, and board cannot rely on con- 
clusory statements when dealing with particu- 
lar combinations of prior art - and specific 
claims, but must set forth rationale on which 
it relies. 

[4] Procedure — Court of Appeals for the 
Federal Circuit (§ 410.03) 

Procedure — Judicial review — Stan- 
dard of review — Patents 
(§ 410.4607.09) ; ■ 

U.S. Court of Appeals for the. Federal Cir- ■ 
cuit will not consider .proposed- alternative 
grounds for affirming decision of Board : of 
Patent Appeals and Interferences rejecting 
patent application for obviousness, since alter- 
native grounds were made at oral argument 
and constitute post hoc rationalization for 
agency action, consideration of which would 
deprive aggrieved party of fair opportunity , to 
support its position. . 



Appeal from the U.S. Patent and Trademark 
Office, Board of Patent Appeals and Interfer- 
ences." ' ~ \ ' - * r 

Patent application of Sang-Su Lee, serial 
no. 07/631,210, directed to method of auto- 
matically displaying functions of video dis- 
play device and demonstrating how to select 
and adjust functions to facilitate user ; -re- 

snnrKP Annlirant o-nrtAale from oi"««- n« 

holding rejection of all claims for obvious- 
ness, and from reaffirmation of that decision 
on reconsideration. Reversed and remanded. 

Richard H. Stern and Robert E. Bushnell, 
Washington, D.C., for Sang Su Lee. 

Sidney O. Johnson Jr., associate solicitor, 
John M. Whealan, solicitor, and Raymond T. 
Chen, Maximilian R. Peterson, and Mark 
Nagumo, associate solicitors, Arlington, Va., 
for Director of U.S. Patent and Trademark Of- 
fice.' " ' 

Before Newman, Clevenger, and Dyk, cir- 
cuit judges. 

Newman, J. 

. Sang-Su Lee appeals the decision of the 
Board of Patent Appeals and Interferences of 



the United States Patent and Trademark Of- 
fice, rejecting all of the claims of Lee's patent 
application Serial No. 07/631,210 entitled 
"Self-Diagnosis and .Sequential-Display 
Method of Every Function." 1 We vacate the 
Board's decision for failure to meet the adju- 
dicative standards for review under the Ad- 
ministrative Procedure Act, and remand for 
further proceedings. ' 

The Prosecution Record 

- Mr. Lee's patent application is directed to a 
method of automatically displaying the func- 
tions of a video display device and demon- 
strating how to select and adjust the functions 
in order to facilitate response by the user. The 
_ display: and demonstration are achieved using 
computer-managed electronics/ "including 
pulse^ width modulation and aW-fine-tuning 
pulses, in accordance witfr procedures de- 
scribed in the specification. Claim 10 is repre- 
sentatives r -V-. • 

10. A method for automatically display- 
ing functions of a video display device, 
comprising: 

determiiiing if a demonstration mode 
is selected; 

if said demonstration mode, is selected, 
automatically entering a picture adjustment 
mode having a picture menu screen display- 
ing a list of a plurality of picture functions; 
■ and ' V . : . 

; * automatically demonstrating selection 
and adjustment of individual ones of said 
plurality of picture functions. 
The examiner- rejected the claims on the 
ground of obviousness, citing the combination 
of two references: United States Patent No. 
4,626,892 to Nortrup, and the Thunderchop- 
per Helicopter Operations Handbook for a 
video game. The Nortrup reference describes 
a television set having a menu display by 
which the user can adjust various picture and 
audio functions; however, the Nortrup display 
does not include a demonstration of how to 
adjust the functions. The Thunderchopper 
Handbook describes the Thunderchopper 
game's video display as having a "demonstra- 
tion mode" showing how to play the game; 
however, the Thunderchopper Handbook 
makes no mention of the adjustment of picture 
or audio functions. The examiner held that it 



1 Ex parte Ue, No. 1994-1989 (Bd Pat. App. & Int 
Aug. 30, 1994; on reconsid'n Sept. 29, 1999). 
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would have been obvious to a person of ordi- 
nary skill to combine the teachings of these 
references to produce the Lee system. 

Lee appealed to the Board, arguing that the 
Thunderchopper Handbook simply explained 
how to play the Thunderchopper game, and 
that the prior art provided no teaching of mo- 
tivation or suggestion to combine this refer- 
ence: with Nortrup, or that such combination 
would produce the Lee invention. The Board 
held that it was not necessary to present a 
source of a teaching; suggestion, or motiva- 
tion to combine these references or their 
teachings. The Board stated: 

.The conclusion of obviousness., may be 
made from common knowledge and com-, 
mon sense of a person of ordinary skill' in 
the art without any specific hint or sugges- 
tion in a particular reference. ' ; " 
Board op. at 7. The Board did not explain the 
, "common knowledge and common sense" on 
which it relied for its conclusion that "the 
-combined teachings of Nortrup and Thunder- 
chopper would have suggested the claimed in- 
vention to those of ordinary skill r in the art. " 

Lee filed a request for reconsideration, to 
which the Board responded after five years. 
The Board reaffirmed its decision, stating that 
the Thunderchopper Handbook was "analo- 
gous art" because it was "from the same .field 
- of endeavor" as the Lee invention, and that 
the field of video games was "reasonably per- 
tinent" to the problem of adjusting display 
functions because the Thunderchopper Hand- 
book showed video demonstrations . of the 
''features" of the game. On the matter of mo- 
tivation to combine the Nortrup and Thunder- 
chopper references, the Board stated that "we 
maintain the position that we stated in our 
prior decision", and that, the Examiner's An- 
swer provided "a well reasoned discussion of 
why there is sufficient motivation to combine 
the references." The Board did not state the 
examiner's reasoning, and review of the Ex- 
aminer's Answer reveals that the examiner 
merely stated that "both the Nortrup function 
menu and the Thunderchopper. demonstration 
mode are program features and that the Thun- 
derchopper mode "is user-friendly" and it 
functions as a tutorial, and that it would have 
been obvious to combine them. 

Lee had pressed the examiner during pros- 
ecution for some teaching, suggestion, or mo- 
tivation in the prior art to select and combine 



the references that were relied on to show ob- 
viousness. The Examiner's Answer before the 
Board, plus a Supplemental - Answer, stated 
that the combination of Thunderchopper with 
Nortrup "would have been obvious to one of 
ordinary skill in the art since the demonstra-* 
tion mode is just a programmable feature 
which can be used in many different devicefs] 
for providing automatic introduction by add- 
ing the proper programming software," and 
that "another motivation would be that the au- 
tomatic demonstration mode is user friendly 
and it functions as a tutorial;" The Board 
adopted the. examiner's answer, stating "the 
examiner has provided a well reasoned discus- 
sion of these references and how the combi- 
nation of these references meets the claim 
limitations." However, perhaps ^recognizing 
that the examiner had " provided insufficient 
justification to, support combining the Nortrup 
and Thunderchopper references, the Board 
held, as stated supra, that a "specific hint or 
suggestion" of motivation to combine was riot 
required . > ' . 

This appeal followed. 

Judicial Review 

Tribunals of the PTO are governed by the 
Administrative Procedure Act, and their rul- 
ings receive the same judicial deference as do 
tribunals , of other administrative agencies. 
Dickinson v. 'Zurko, 527 U.S. 150, 50 USPQ2d 
1930 (1999). Thus, on appeal we review a 
PTO Board's findings and conclusions in ac- 
cordance with the following criteria: - 

5 U.S.C. §-706(2) The reviewing court 
shall — 

. (2) hold unlawful' and set aside agency 
actions, findings, and conclusions found 
to be — , 

(A) arbitrary, capricious, an abuse of dis- 
cretion, or otherwise' not in accordance 
with law-, 

* *,* * 

(E) unsupported by substantial evidence 
in a case subject to sections 556 and 557 
of this title or otherwise reviewed on the 
record of an agency hearing provided by 
statute; , 



For judicial review to be meaningfully 
achieved within these strictures, the agency 
tribunal must present a -fuJI and reasoned ex- 
planation of its decision. The agency tribunal 
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must set forth its findings and the grounds 
thereof; "as supported by the agency record, 
and- explain its application of the law to the 
found facts. The Court has often explained: 

the Administrative Procedure Act, which 
governs the proceedings of administrative 
agencies and related judicial review, estab- 
lishes a scheme of "reasoned decisionmak- 
ing." Not only must an -agency's decreed 
result be within the scope of its lawful au- 
thority, but the process by which it reaches 
that result must be logical and rational; 
Allentown Mack Sales and Service, Inc. v. Na- 
tional Labor Relations Bd, 522 U.S. 359,. 374 
(1998) (citation omitted). This standard re- 
quires" that the agency not only have reached a 
sound decision,- but have articulated the rea- 
sons for that decision. The reviewing court. is 
thus enabled to perform meaningful review 
within "the strictures of the APA, for the court 
will have "a basis on which ' to determine 
"whether the decision was based on the rel- 
evant factors and whether there has been a 
clear error of judgment." Citizens to Preserve 
Overton Park v. Volpe, 401 U.S. 402, 416 
(1971). Judicial review of a Board decision 
denying an application for patent is thus 
founded on the" obligation of the agency to 
make the necessary findings and to provide an 
adrninistrative record showing the evidence 
on which the findings are based, accompanied 
by the agency's reasoning in reaching its-con- 
clusions. See In re Zurkq, 258 F.3d 1379, 
1386, 59 USPQ2d, 1693,-1697 (Fed. Cir. 
2001) (review is on the administrative record); 
In re Gartside, 203 F.3d 1305, 1314, 53 
USPQ2d 1769, 1774 (Fed. Cir. 2000) (Board 
decision "must be justified within the four 
comers of the record"). . 

[1] As applied to the determination of pat- 
entability vel hon when the issue is obvious- 
ness, "it is fundamental that rejections under 
35 U.S;C. § 103 must be based oh evidence 
comprehended by the language of that sec- 
tion." In re Grasselli, "1 13-F.2d 731, 739, 218 
USPQ 769, 775 (Fed: Cir. 1983). The essen- 
tial factual evidence on the issue of obvious- 
ness is set forth in Graham v. John Deere Co., 
383 U.S. 1, 17-18, 148 USPQ 459, 467 (1966) 
and extensive ensuing precedent. The patent 
examination process centers on prior art and 
the analysis thereof. When patentability turns 
on the question of obviousness, the search for 
and analysis of the prior art includes evidence 
relevant to the finding of whether there is a 



teaching, motivation, or suggestion to select 
arid combine the references relied on as evi- 
dence of obviousness. See, e.g., McGinley v. 
Franklin Sports, Inc., 262 F.3d 1339, 1351-52, 
.60 USPQ2d 1001, 1008 (Fed. Cir. 2001) (*-'the 
central question is whether there is reason to 
combine [the] references," a question of fact 
drawing on the Graham factors).. 

"The factual inquiry whether to combine - 
references must be thorough and searching." 
Id. It must be based on objective evidence of 
record. This precedent has been reinforced in 
myriad decisions, ' and cannot be dispensed 
with. See, e.g., Brown & Williamson Tobacco 
Corp. v. Philip Morris Inc., 229 F.3d 1120, . 
1124-25, 56 USPQ2d 1456, 1459 (Fed. Cir. 
2000) ("a showing of a suggestion, teaching, 
or - motivation to combine the prior art refer- 
ences is an 'essential component of an obvi- 
ousness holding' ") (quoting CR. Bard, Inc., 
v. M3 Systems] Inc.; 157 F.3d 1340, 1352, 48 
USPQ2d 1225, 1232 (Fed. Cir. 1998)); In re 
Dembiczak, 175 F.3d 994, 999, 50 USPQ2d 
1614, 1617 (Fed. Cir. 1999) (".Our case law 
makes clear that the best defense against the 
subtle* but powerful attraction of a hindsight- 
based obviousness analysis is rigorous appli- 
cation of the requirement for a showing of the 
teaching or motivation to combine prior art 
references."); In re Dance,; 160 F.3d 1339, 
1343, 48 USPQ2d 1635, 1637 (Fed. Cir. 
1998) (there must be some motivation,, sug- 
gestion, or teaching of the desirability of mak- 
ing the specific combination that was made by 
the applicant); In re Fine, 837 F.2d 1071, 
1075, 5 USPQ2d 1596, 1600 (Fed. Cir. 1988) 
(" 'teachings of references can be combined 
only if there is. some suggestion or incentive 
to do so/ ") (emphasis in original) (quoting 
ACS Hosp. Sys., Inc. v. Montefiore Hosp., 732 
F.2d 1572,- 1577, 221 USPQ 929, 933 (Fed. 
Cir. 1984)). 

The need for specificity pervades this au- 
thority. See, e.g., In re Kotzab, 111 F.3d 1365, 
1371, 55 USPQ2d 1313, 1317 (Fed. Cir. 
2000) ("particular findings must be made as 
to the reason the skilled artisan, with no 
knowledge of the claimed invention, would 
have selected these components for combina- 
tion in the manner claimed"); In re Rouffet, 
149 F.3d 1350, 1359, 47 USPQ2d 1453, 1459 
(Fed. Cir. 1998) ("even when the level of skill 
in the art is high, the Board must identify spe- 
cifically the principle, known to one of ordi- 
nary skill, that suggests the claimed combina- 
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tion. In. other words, the Board must explain 
the reasons one of. ordinary skill in, the art 
would have been motivated to select the refer- 
ences and to combine, them to render' the 
claimed invention obvious."); In re Fritch, 
972 F2d 1260, 1265, 23 USPQ2d 1780, 1783 
(Fed. Cir. 1992) (the examiner can satisfy the 
burden of showing obviousness of the combi- 
nation , "only by showing some . objective 
teaching, in the prior art or that knowledge 
generally available to one of ordinary skill in 
the art would lead that individual to combine 
the relevant teachings of the references"). 

[2] With respect to Lee's application, nei- 
ther the examiner nor the Board adequately 
supported, the selection and combination of 
the Nortrup and Thunderchopper references to 
render obvious that which. Lee described. The 
examiner's conclusory statements that "the 
demonstration ' mode is just a programmable 
feature which can be used in many different 
device [s] for providing automatic introduction 
by adding the proper programming software" 
and that "another motivation would be that 
the automatic demonstration mode is user 
friendly and it functions as a tutorial", do not 
adequately address the issue of motivation to 
combine. This factual question of .motivation 
is material to patentability, and could not be 
resolved on subjective belief and unknown au- 
thority. It is improper, in determining whether 
■ a person of ordinary skill would have been led 
to this combination of references, simply to 
"[use] that which the inventor taught against 
its teacher." W.L Gore v. Garlock; Inc., 721 
F.2d 1540, 1553, 220 USPQ 303, 312-13 
(Fed. Cir. 1983), Thus the Board must not 
only assure that the requisite findings are . 
made, based on evidence of record, but must 
also explain the reasoning by which the find- 
ings are deemed to support the agency's con-, 
elusion. 

Deferential judicial review under the Ad- .. 
ministrative Procedure Act does not relieve 
the agency of its obligation to develop an evi- 
dentiary basis for its findings. To the contrary, 
the Administrative Procedure Act reinforces 
this obligation. See, e.g., Motor Vehicle Manu- 
facturers Ass 'n v. State Farm Mutual Automo- 
bile Ins. Co., 463 U.S. 29, 43 (1983) ("the 
agency must examine the relevant data- and ar- 
ticulate a satisfactory explanation for its' ac- 
tion including a 'rational connection between 
the facts found and the choice made.' ") 
(quoting Burlington Truck Lines v. United. 



States, 371 U.S. 156, 168 (1962)); Securities 
& Exchange Comm'n y. Chenery Corp., 318 
U.S. 80, 94 (1943) ("The orderly function of 
the. process of review, requires that the 
grounds upon which the administrative 
agency acted are clearly disclosed and ad- 
equately sustained."). 

In its , decision on Lee's patent application, 
the. Board rejected the need for "any specific 
hint or suggestion, in a particular, reference'' to 
support the combination of the Nortrup and 
Thunderchopper references. . Omission of a 
relevant factor required by precedent is both 
legal error and arbitrary agency action: See 
Motor Vehicle Manufacturers, 463 U.S. at 43 
("an agency rule would be arbitrary and capri- 
cious if the agency . .. . entirely failed to con- 
sider an important aspect of the problem"); 
Mullins v. Department of Energy, 50 F.3d 990, 
992. (Fed. Gir. 1995) ("It is well established 
that agencies have a duty to provide review- 
ing courts with a sufficient explanation for 
their decisions so that those decisions may be 
judged against the relevant statutory stan- 
dards, and that failure to provide such an. ex- 
planation is grounds for striking down the ac- 
tion."). As discussed in National Labor Rela- 
tions Bd. v. Ashkenazy Property Mgt. Corp., 
817 F.2d 74, 75 (9th "Cir. 1987), an agency is 
"not free to refuse to follow circuit prece- 
■ dent." . 

The foundation of the principle of judicial 
deference to the rulings of agency tribunals is 
that the tribunal has" specialized knowledge 
and expertise, such that when reasoned find- 
ings are m2.de,, a reviewing court may confi- 
dently defer to the agency's application of its 
knowledge in its area of expertise. Reasoned 
findings - are critical to the performance of 
agency functions and judicial reliance on 
agency competence. See Baltimore and Ohio 
R. R. Co. v. Aberdeen &.. Rockfish R. R. Co., 
393 U.S. 87, 91-92. (1968) (absent reasoned 
findings based on substantial evidence effec- 
tive review .would become lost "in the haze of 
so-called expertise"). The "common knowl- 
edge and common sense" on which the Board 
relied in rejecting Lee's application are not the 
specialized knowledge and expertise contem- 
plated by the Administrative Procedure Act. 
Conclusory statements such as those here pro- 
vided do not fulfill the agency's' obligation. 
This court explained in Zurko, 258 R3d at 
1385, 59,USPQ2d at 1697, that "deficiencies 
of the cited references cannot be remedied by 
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the Board's general conclusions about what is 
'basic knowledge' or 'common sense/. ".The 
Board's findings must extend to all material 
facts and must be documented on the record, 
lest the "haze of so-called expertise" acquire' 
insulation from accountability. "Common 
knowledge and common sense," even if as- 
sumed to derive from the agency's expertise, 
do not substitute for authority when the law 
requires authority. See Allentown Mack, 522 
U.S. at 376 ("Because reasoned decisionmak- 
ing demands it, and because the systemic con- 
sequences of any other approach are unaccept- 
able, the Board must be required to apply in 
fact the clearly understood legal standards; that 
it enunciates in principle : . . .") 

The case on which the Board relies for its 
departure from precedent, In re. Bozek, 416 
R2d 1385, 163.USPQ 545 (CCPA 1969), in- 
deed mentions "common knowledge and 
common 1 sense," the CCPA. stating that the 
phrase was used by the Solicitor to support the 
Board's conclusion of obviousness based on 
evidence in the prior art. Bozek did .not hold 
that common knowledge and common sense 
are a substitute for evidence, but only that 
they.' may be .applied to analysis of the evi- 
dence. Bozek did not hold that objective 
analysis, proper authority, and reasoned find- . 
ings can be omitted from Board decisions. 
Nor does. Bozek, after thirty-two years of iso- 
lation, outweigh the dozens of rulings of the 
Federal Circuit and the Court, of Customs and 
Patent Appeals that determination of patent- 
ability must be based on evidence. This court 
has remarked, in Smiths Industries Medical 
Systems, Inc. v. Vital Signs, Inc., 183 F.3d 
1347, 1356, 51 USPQ2d 1415, 1421 (Fed. Cir. 
1999), that Bozek' s reference to common 
knowledge "does not in and of itself make it 
so" absent evidence of such knowledge. 

[3] The determination of, patentability on 
the ground of unobviousness is-ultimately one 
of judgment. In furtherance" of the judgmental ' 
process, the patent examination procedure 
serves both to find, and to^ place on the official 
record, that which has been considered with 
respect to patentability. The patent examiner 
and the Board are deemed to. have experience 
in the field of the invention; however, this ex- 
perience, insofar as applied to the determina? 
tion of patentability, must be applied from the 
viewpoint of "the person having ordinary skill 
in the art to which said subject matter per- 
tains," the words of section 103. In finding the 



relevant facts, in assessing the significance of 
the prior art, and in making the ultimate deter- 
mination of the issue of obviousness, the ex- 
aminer and the Board are presumed to act 
from this viewpoint Thus when they rely on 
what they assert to be general knowledge to 
negate patentability, that knowledge must be 
articulated and placed on the record. The fail- 
ure to do so is not consistent, with either effec- 
tive administrative procedure or effective judi- 
cial review. The board cannot rely on conclu- 
sory statements when dealing with particular 
combinations of prior art arid specific claims, 
' but must set forth the rationale on which it re- 
lies. - 

Alternative Grounds 

[4] At oral argument the PTO Solicitor pro- 
posed alternative grounds on which this court 
might affirm the Board's decision. However, 
as stated in Burlington Truck Lines/ Inc. v. 
United States, 371 U.S. 156, 168 (1962), 
"courts may not accept appellate counsel's 
post hoc rationalization for agency action." 
Consideration by the appellate tribunal of new 
agency justifications deprives the aggrieved 
party of a fair opportunity to support its posi- 
tion; thus review of an administrative decision 
must be made on the grounds relied on by the 
agency. "If those grounds are inadequate or 
improper, the court is powerless to affirm the 
administrative action by substituting what it 
. considers to be a more adequate or proper ba- 
sis." Securities & Exchange Cbmm'n v.Chen- 
ery Corp., 332 U.S. 194, 196 (1947). As reit- 
erated in Federal Election Comm'n v. Akins, 
524 U.S.- 11, 25 (1998), "If a reviewing court 
agrees that the agency misinterpreted the law, 
it will set aside the agency's action and re- 
mand the case — '• even though .the agency 
' (like a new jury after a mistrial) might later, 
in the exercise of its lawful discretion, reach 
the same result for a different reason." Thus 
we decline to consider alternative grounds that 
might support the Board's decision. 

Further Proceedings _ 

Sound administrative procedure requires 
that the agency apply the law in accordance 
with statute and precedent. The agency tribu- 
nal must make findings of relevant facts, and 
present its reasoning in sufficient detail that 
the court may conduct meaningful review of 
the agency action. In Radio-Television News 
Directors Ass'n u FCC, 184 R3d 872 (D.C. 
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Cir. 1999) the court discussed the. "fine line 
between agency reasoning that, is 'so crippled 
as to be unlawful' and action that is potenT 
tially lawful but insufficiently or inappropri- 
ately, explained," quoting from Checkoskyv.. 
Securities & Exch. Comm'n, 23 F.3d 452, 464 
(D.C. Cir. 1994); the court explained that 
"[i]n the former circumstance, the court's 
practice is to vacate the agency's order,, while 
in the latter the court frequently remands for 
further explanation (including, discussion of 
the relevant ' factors and precedents) while 
withholding judgment on the lawfulness of the 
agency's proposed action." Id at 888. In this 
case the Board's analysis of the Lee .invention 
does not comport with either the legal require- 
ments for determination of obviousness oV 
with the requirements of the Administrative 
Procedure Act that the agency tribunal set- 
forth [he findings and explanations needed for. 
"reasoned., decisionmaking." ..Remand/, for 
these purposes is required. See Overton Park, 
401 U.S. at .420-221 (remanding for further- 
proceedings appropriate to the administrative 
process). . 

VACATED AND REMANDED .. .. 



instructions, represent mere unprotected facts 
or protectable expression. 



JUDICIAL PRACTICE 
PROCEDURE 



AND 



[2] Procedure — Limitations period; time-"' 1 
liness (§410.05). 
Plaintiffs' claim for copyright infringement 
is not. barred by three-year statute of limita- 
tions, specified by 17 U.S.C, § 507(b), even- 
though infringement claim was brought more 
than three years after infringing work was first 
published, since discovery rule and other eq- 
uitable tolling doctrines' apply to copyright 
claims, since plaintiffs' cause of action argu- • 
ably, did not accrue until they discovered de- • 
fondants' book, less than one year before suit 
was brought, and since even if claim accrued 
on date of first publication, limitations period 
bars only remedy, not substantive right. ■ 



Barbour v. Head 

U.S. District Court 
Southern District of Texas 

.No. G-0 1-491 . . 
Decided December 21, 2001 

COPYRIGHTS 

[1] Non-copyrightable matter — Ideas and 
" systems (§ 211.05) 

Defendants are not entitled to. summary 
judgment that plaintiffs' cooking recipes are 
uncopyrightable, even though .17 U.S.C 
§ 102(b) denies copyright protection to mere 
procedures or processes, since neither courts 
nor Register of Copyrights have declared that 
recipes are per se uncopyrightable, since de- 
fendants have not shown that plaintiffs' cook, 
book is copyrighted' as factual compilation or' 
collective work rather than literary work, and 
since even if book is not literary work, genu- ' 
ine issue of material fact exists as to whether 
plaintiffs' recipes, which contain more than 
mechanical listings of ingredients and cooking 



Action by Judy Barbour and Cookbook Re- 
sources LLC against James Head and Penfield 
Press Inc. for copyright infringement, ' and for 
unfair competition through misappropriation 
and conversion. On defendants' motion for 
summary judgment. Denied as to copyright ' 
claims; granted as to state law claims. 

G.P Hardy III, Houston, Texas, for plain- 
tiffs:' •> 

Karen Bryant Tripp, Houston, for defen- 
. dants. 

Kent, J. 

ORDER GRANTING IN PART DEFEN- 
DANT: PEN FIELD PRESS' MOTION TO 
DISMISS. 

This case involves a rustled cowboy cook- 
book. On August 13, 2001, Plaintiffs Judy 
Barbour ("Barbour") and Cookbook Re- 
sources,:; L.L.C. ("Cookbook Resources") 
filed causes of action for copyright infringe- 
ment, unfair competition through misappro- 
priation, and conversion, with which they're 
fixin' to brand Defendants James Head 
("Head") and Penfield Press, Inc. ("Penfield 
Press"). On October 25, 2001, to bust out of 
the corral, Defendant Penfield Press filed a 
Motion to Dismiss pursuant to Fed.R.Civ.R 
12(b)(6). For the reasons articulated below, 
Defendant's Motion to Dismiss shall be 
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mately 80% of the total amount sought for 
that period. 

f' [6] A total award of $130,643.75 for attor- 
neys' iees is appropriate in this case in light 
of the KirribrelVs factors that apply. Prep- 
aration for the trial with respect to this 
particular defendant involved several, pre- 
trial motions, several raising coihjpl ex factual 
questions, and the .fast pace of this docket 
required an intense dedication of attorney 
. time to the, litigation. :(Factbrs l; 2 and 7). 
Moreover, the Court notes that, plaintiffs' 
■: counsel 'took .the case on a' contingent- fee 
basis; which means that given the amount of 
the judgment, they will be made whole forall 
work expended in this case. We" also' point 
out; that the actual trial was relatively shbr.t, 
partly as a result of the defendant's having so 
little evidence to present. Finally, in light of 
,the damages awarded, $130,643.75 is not an 
unreasonably high attorneys' fee. (Factor 8 
and 12). . 

With respect to costs, the Court finds it 
appropriate, for the same reasons discussed 
above, to award the plaintiffs the :,total 
amounts reflected only on the law firm's 
invoices dated March 26, 1.998 ($37,612.84) 
and April 9, 1-998 ($1.7,309.51), for a total 
; award of $54,922.35. This amount includes 
allcbsts incurred from February 23, 1998, to 
■April 9, 1998. ' : ->r. , 

IV '. Conclusion . : , v 

For the reasons stated in open court and in 
this Memorandum Opinion, BRFG's Motion 
for Declaration of Mistrial and Motion for a 
JNOV or in the Alternative a New Trial will 
be denied. In addition, plaintiffs' Motion' 6f 
Trebled Damages arid Attorneys' Fees will 
■. be denied as< to the trebled damages and 
granted as to the attorneys' fees" and costs, 
and. the Court will award plaintiffs 
$130,643.75 in " attorneys' fees' and 
$54,922:25 in costs. An appropriate order 

Will iSSUe. ' :r^H:r>, 

- - The Clerk is directed to forward copies of 
this 'Memorandum Opinion to counsel' of. 
record.. •."•■-."■-;* -.- 



$54,922.35 in costs, for a total award of 
$185,566.10. 

The Clerk is directed to forward copies of 
this Order to counsel of record. 
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In re Rouffet 

No. 97-1492 ; v- -' 
Decided July. 15, 1998 • 

PATENTS •' . 

1. Patentability/Validity ^ Obviousness — 
Combining references (§115.09.05) - ' 

'"' Claimed low Orbit satellite communica- 
tions system for mobile terminals, Avhich 
addresses problem of minimizing "hand- 
over" of receiver from beam footprint of one 
transmitting . satellite to that of another 
through use of multiple fan-shaped beams, is 
not prima facie obvious over combination of 
three prior art references, since critical refer- 
ence that teaches use of. fan-shaped . beam to 
transmit; from ground station to orbiting sat- 
ellites does not specifically address, handover 
minimization, and to extent it .addresses 
handover problem at all, does so with orbit 
selection rather than beam shape, and since 
there is no reason one of ordinary skill in art, 
seeking to minimize handovers due: ta-saiei- 
lite motion, would have been motivated to 
combine this reference with remaining refer- 
ences in manner that would render .claimed 
invention obvious. . j .:\V,-ii 

2. Patentability /Validity — Obviousness f- 
Person of ordinary; skill in iart 



[\\.\ order 

u- For the reasons stated in an accompanying 
■Memorandum Opinion, SRFG's Motion for 
Declaration of Mistrial and Motion , for a 
JNOV or in the Alternative a New Trial are 
DENIED, and plaintiffs' Motion for Trebled 
Damages and Attorneys'. Fees is: GRANT- 
ED IN PART as to attorneys' fees and costs 
and DENIED as to trebled damages, and it 
is hereby. 

•ORDERED that plaintiffs be and- are 
awarded $130,643.75 in attorneys'. fees ;and 



Patentability /Validity — Obviousness 
Combining references (§115.0905), 

Three possible sources for rhptivation to 
combine prior art references to" mariner that 
would render claimed invention obvious are 
nature'of problem to be solved, teachings of 
prior art, and knowledge of persons of ordi- 
nary skill in art; high level of skill in- field of 
art Cannot, be relied upon to pirovide neces- 
sary; motivation- absent explanation of what 
specific understanding or technical principle, 
within knowledge one of ordinary skill in art, 
would have suggested' combination, since, if 
such rote invocation could suffice to supply 
motivation to combine, more sophisticated 
scientific fields would rarely, if ever, exper- 
ience patentable technical advance. 




3. Paten tability /Validity ..^ Obviousness 
Person of ordinary skill in art 
.(§1 15.0902) : — • 

Patentability /Validity — Obviousness — 
Combining references (§115.0905) „ 

Claimed low orbit satellite communica- 
tions system for mobile terminals is. not 
prima facie obvious over, combination of two 
prior art references,- even though person pos- 
sessing high level of skill characteristic of 
this field would know to account for differ- 
ences between claimed invention and prior 
art combination, since high level of skill in 
art, without more, cannot ..supply required 
motivation to combine references, arid does 
not overcome absence of any actual, sugges- 
tion to combine; obviousness rejection will 
not be- upheld,. even where skill in art is high, 
absent" specific" identification of principle, 
known to. one of ordinary skill,, that suggests 
claimed combination. . , . >rv. v '' w.:- 



Appeal from, the U.S. : Patent and Trade- 
mark/Office; Boafd ; of Patent Appeals and 
interferences: ' "--"•v ■■■■ ~ *n 

Patent application of Denis Rouffet,' Yan- 
riick Tanguy* and Frederic Berthault, serial 
no. 07/888,791 piled May 27, 1992; -From 
decision upholding examiner's final rejection 
of .application 'as obvious under 35 *;USC 
103(a)i r applicants appeal. Reversed/ - 

Richard C Turner and Grant K. Rowan, of 
ISughfuev-Mion^Ziri Macpeak &Seas, 
■;Wa.shington,;D;C:i for appellants: ; ■ • 

David J. Ball Jri, associate solicitor,. Nancy 
J. Linck,- solicitor, Albin F. Drost, deputy 
solicitor, Craig R. Kaufman v as.sociate,sor 
. . iicitor, "and Scott. A.! Chambers, associate 
•solicitor, U.S. Patent and Trademark Of- 
fice, Arlington, Va., for appellee. 

Before Piagef," circuit^ judge, Archer, senior 
circuit Judge, and Rader; circuit judge. 

Rader, J.; ;r . -^y.v' 

Denis Rouffet, Yannick Tanguy, and .Fre- 
deric Berthault (collectively, RoufFet) sub- 
mitted application^ 07/888,79.1 (the. applicav 
tion).on May 27, 1992. The Board of. Patent 
Appeals and . Interferences; . (the Board) afV 
firmed final rejection of -the application as 
' obvious, under . 35 U.S/C, §. 103(a), See Ex 
parte. Rouffet^o. 96-1553 (Bd. Pat. App. 
&• Int. Apr. -I.6y 1997). Because the. Board 
reversible erred in identifying a motivation 
to combine the references, this court revers- 
es. 



.... I. '■• V. ..... v; 

Satellites in a . geosynchronous or geo- 
stationary orbit remain oyer the same point 
on the Earth's surface. Their constant, posi- 
tion above the Earth's surface facilitates 
communications. These satellites project a 
number.of.beams.'to the Earth. jEach.^be^ni 
transmits to its area of coverage, or footprint, 
on the. Earth's surface.' In order to = provide 
complete Coverage, adjacent 'footprints over- 
lap slightly and therefore rrhust use.different 
frequencies to" avoid interference. ; However, 
two, pr more ;nqn-overlapping footprints' can 
use 'the same set. of frequencies, in order to 
use ^efficiently *' the .limited iradib . spectrurri; 
Figure IJ'irbm the application shows' the 
coverage! of. a portion of .the ;Earth's _ siir-face 
provided by . multiple cone shaped beams: .. :. 



FIG. 1 




Frequency reuse ^techniques, however, 
have a limited ability, to . compensate for. 
congestion in geostationary orbits. To allevi- 
ate, the. orbit congestion problem, ' new tele- 
communications systems use a network of. 
satellites in low Earth orbit. When viewed . 
from a . fixed point an the Earth's surface, 
such satellites do not remain stationary but 
move overhead. A satellite's motion as: it- 
transmits a plurality of cone-shaped beams 
creates a new problem. The satellite's move- 
ment causes a receiver on the Earth's surface 
to move from the footprint of. one beam into a 
second; beam transmitted by the same satek 
lite. Eventually, 1 the satellite's motion causes- 
the receiver ; to*move from': the footprint of a- 
beam transmitted by one satellite into the 
footprint of a beam transmitted by a -second 1 
satellite. Each switch from one footprint to 
another creates a "handover"^ event analo- 
gous to that which occurs when a traditional 
cellular phone travels frorri one cell" to an- 
other^ -Handovers are undesirable because 
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they can cause interruptions in signal .trans- 
mission and reception. : .' . " 
. Rouffet's application discloses technology 
to reduce the number of handovers, between 
beams transmitted by the same satellite; In 
particular, Rouffet eliminates handovers 
caused * solely by the . satellite's motion; To 
accomplish this goal* Rouffet changes the 
shape of the beam transmitted by. the. satel- 
lite's antenna, ^Rouffet's satellites transmit 
fail-shaped beams. A fan.beamhas.an ellipti- 
cal footprint; Rouffeti aligns the long axis of 
his beams parallel to the direction; of the 
satellite's -motion across the. Earth's surface, 
By elongating the. beam's footprint "in the 
direction of satellite travel, Rouffet's inven- 
tion ensures that a fixed point on the Earth's 
surface likely will remain within a single 
footprint until it is necessary -to = switch to 
another , satellite. Because Rouffet r s inven- 
tion does not address , handovers caused ;by 
the .motion of the receiver across the Earth's 
surface, his. arrangement reduces, b.ui does 
not eliminate,: handovers. Figure 3; : from the 
application shows the footprints 12 from six 
beams, aligned in the direction of satellite 
motion 15: ; ^ : - >V'-\ii • .... 



' .'rAG.,?.-^'"'''. 




■■ The application contains ten claims: that 
stand or . fall as a group. Claim; is 
representative: ; -r- - -^. ---h^A 

. : ; A ; ; low orbit ^ satellite. . eommun ica t ions 

'system for mobile terminals, -wherein the 
.. communications antenna system, of each 

V satellite provides. isoflux coverage made up 
of a plurality of fan beams that are eIon : 
' gate ia-the travel direction: ofthe satellite. 

The. .examiner initially . rejected Rouffet's 
claims as unpatentable .over U.S.. Pat. No! 
5,199,672 (King) in view of U.S. Pat. ;No. 
4,872,015 (Rosen) anaVa conference report 
entitled "A Novel Non-Geostationary Satel : 
lite Communications System,". Conference 
Record, International Conference on Com- 
munications, 198.1 (Ruddy). On. appeal to 
the Board,, the examiner added ari alternar 



tive. ground ..for rejection, holding that the 
claims ^were obvious . over U.S. Pat. No. 
5,394;56.h (Freeburg) in view of U.S. Pat. 
No. 5,170,485" (Levine).' 

• On April T 6,. 1997, the -Board issued. its 
decision. Because. Rouffet had specified that 
the- claims would stand or fall.as:-a. group 
based; .Von the patentability of claim 1, .the 
Board limited its opinion to that; claim. The 
Board; unanimously determined that the^ ex- 
aminer- had properly rejected claim as 
obvious, over King in view, of Rosen -and 
Ruddy. The Board, on a split vote, : also 
affirmed the. rejection over, Freeburg in view 
of Levine, - 

• ; To ieject claims in an application under 
seetion?103, an examiner rriust showman un r 
rebutted prima facie case of obviousness. See 
In re Deuel, 5i.-F.3d 1552,1557, 34 USPQ2d 
1210, 1214 (Fed. . Cir. 1995).. In .the absence 
of a proper prima facie case of obviousness, 
an applicant who complies with the . other 
statutory requirements is entitled to a patr 
ent: See Imre Oetiker y 977 F.2d 1443, 1445, 
2.4USPQ2dT443, 1444 (Fed. Cir. 1992). On 
appeal to the. Board, an applicant can dyer- 
come a rejection by. showing insufficient evi- 
dence oi.prima facie obviousness or by 'rebutr 
ting the prima facie case with evidence of 
secondary . indicia of nonobyiousness. See id. 
-,j= While this court reviews the Board's deter^ 
mina tion in Tight of the ..entire record,, an 
applicant may specifically challenge an:obyi-: 
ousness rejection by showing that the Board 
reached an incorrect conclusion of ^obvious- 
ness or that the Board based its obviousness 
determination on incorrect factual predi- 
cates. This court reviews the ultimate deter- 
mination of obviousness as a question of law. 
See In re Lueders, 111 F.3d, 1569,. 1571, ,42 
USPQ2d;i48 1 ,: 1482 (Fed. : Cir. 1 997),. Tfie 
factual predicates underlying an obviousness 
determination include the scope and content 
of the prior art, the differences .between the 
prior art and the claimed invention^ and, the 
level of ordinary skill in.. the: -art.--: See. JdQlh 
arch Knit i ing Mach . . Corp, , v. Sulzer. Moral 
GmbH,. 139. F;3d,877, v 881, V45 USPQ2d 
1977, 1981, (Fed. Cir. 1998). This court 
reviews the Board's factual, findings; for clear 
error. See In re Zurkoi 142 Ts3d, 1447, 
1449, 46 USPQ2d .1691, 1693 (Fed. Cir. 
1998) (in, banc); -Leuders, ? llL : : JFi3d. : -at 
157i-72. " 'A finding is ..clearly erroneous 
when, although there is evidence: to. support 
it, the reviewing court on the entire evidence 
is left with the definite and firm conviction 
that a. mistake has been committed.'." Fn re 
Graves, 69 F.3d 1147, 11.51, 36 USPQ2d 
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1697, 1700 (Fed. Cir. 1995) (quoting United F.3d at 1573 (citing Ashland Oil, Inc. v. 

States v;' United States Gypsum Co 7 333 Delta Resins & Refractories, Inc., 116 F.2d 

U.S. 364, 395 [76 USPQ 430] (1948)).' :: 281-, 297 n.24, 227 USPQ 6.57, 667 n.24 

The secondary considerations are also es- (Fed, Cir. 1985)). Therefore; " [w]hen deter- 

sential componentsbf the obviousness deter- mining the patentability of a claimed inven- 

minatibri. See In re Erriert, 7 124 F.3d 1458, tion which combines two known elements; 

1462, 44 USPQ2d 1149, 1153 (Fedi Cir. "the question is whether there is something 

1997) ("Without Eniert providing rebuttal m the. prior art as a whole to suggest the 

evidence, this' prima facie case of obvious- desirability^ and thus- : the obviousness, of 

-ness must stand."); This objective evidence making the combination. * " See In re Beat- 

of nonobviousriess includes copying,' long felt tie; 974- F,2d 1309,; 131 l-12 v 24 USPQ2d 

but unsolved heed, failure 6f others, see Gra- 1040, 1042 (Fed. Cir. 1992) (quoting X/m/e- 

h'am v. John Deere Co., 383 U.S. 1, 17-18 mann Maschinenfabrik GmbH v. American 

ri48 USPQ 459] (1966), commercial sue* Hoist & Derrick Co., 730 F:2d 1452, 1462, 

cess, see In re Huang, 100 F.3d 135, 139-40; 221 USPQ 481, 488 (Fed, Cir, 1984)). 

40USPQ2dl685, 1689-90 (Fed. Cir. 1996), ■ ■■ * . :r ' 

unexpected results created by the claimed ; ; ' ' J IJI 
invention, unexpected properties of the 

claimed invention, see In re Mayne, 104 F.3d The parties agree that the five references 

1339, 1342, 41 USPQ2d 1451, 1454 (Fed. asserted by the examiner are in the same 

Cir. 1997); In re Woodruff, 919 F.2d 1575* field of endeavor as the invention. The par- 

1578,- 16 USP02d 4934,;;1936-37 (Fed. Cir. ties also agree that the pertinent level of skill 

1990), licenses showing industry respect for in the art ; — design of satellite communica- 

the invention, see Arkie- Lures, Inc. v. Gene tipns' systems ^ is' high; On appeal, Rouffet 
Larew Tackle, Inc.; 419 F.3d 953,'-957i 43 ' asserts ' that the examiner and the Board 

USPQ2d 1294, 1297 (Fed, Cir. 1997); Pen- erred by improperly combining references to 

tec Inc. v: Graphic Controls Corp -,116 F.2d render the claimed invention obvious. 
30^, 316, 227 USPQ 766, 771= (Fed; Cir. 

1985), and skepticism of skilled artisans be- The Combination of King, Rosen, and 

fore the invention, see In re Dow C hem. Co., Ruddy ' 
837 F.2d 469, 473, 5 USPQ2d 1529, 1532 

(Fed. Cir. 1988). The Board must consider The Board first affirmed the rejection of 

all of the applicant's evidence. See'Oetikef, Rouffet's claims over a combination of King, 

977 F,2d at 1445 ("An observation : byr the Rosen, and Ruddy. King discloses a system 

Board that the examiner made a prima facie for launching /a plurality of satellites into low 

case is not improper, as long as the ultimate Earth orbits Jfrbmii; singte launch vehicle, 

determination of patentability is made on the Rosen teaches /a: geostationary satellite that 

entire Tecordv"); In re Piasecki, 145 F:2d uses a plurality of. fan beams with. their long 

1468; 1472, 223 USF'Q 785, 788 fFed; Cir. axes oriented^in Sn xast-west -direction to 

1984). The court reviews factual conclusions communicateV>yit ti rripbile and- fixed termi- 

drawri from this evidence for clear error. nals on the Earth. ;: 

Whether the evidence presented suffices to The final, and most important, reference 

rebut the prima facie case is part of . the in this combination is Ruddy. Ruddy de- 

ultimate conclusion of obviousness arid is scribes a television broadcast system that 

therefore a question of law. : v n - uses a series of satellites to retransmit signals 

When a rejection depends on a combiha- sent from a ground station over a wide area, 

tion of prior aft- -references Rather than, using a geostationary orbit, 

some teaching; suggestion, or motivation r td Ruddy teaches the use of a series of satellites 

combine the references. : See In re Geiger, in Molriiya orbits. A satellite in a Molniya 

815 F.2d 686, 688, 2 USPQ2d l276, 1278 orbit always follows the same path through 

(Fed. Cir, 1987) /Although the suggestion to the sky when viewed from a fixed point on 

combine references may flow from the na- the ground. Viewed fr6m the Earth, the. 

ture of the problem, see Pro-Mold cfe Tool orbital path includes a harrow, elliptical apo- 

Co: v. Great Lakes' Plastics, Inc., 75 F.3d gee loop. In order to transmit to these moving 

1568, 1573, ^37 U$PQ2d 1626,' 1630 (Fed. satellites from a ground station,. Ruddy uses 

Cir. 1996), the suggestion more often comes a fan beam with a long axis aligned with the 

from the teachings of the pertinent refer- long - axis of the orbit's apogee loop: This 

ences, see In re Sernaker, 702 F.2d 989, 994, alignment places the entire apogee loop with- 

21 7 USPQ 1; '5 (Fed. Cir. 198 3);' or from the in the footprint of the beam and eliminates 

ordinary knowledge of those skilled in the art the need for the ground station's antenna to" 

that certain references are of "special impor- track the satellite's motion around the apo- 

tarice in a particular field, see Pro-Mold, 75 gee loop. Ruddy further teaches orbit param- 
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eters and spacing of multiple satellites, to 
ensure that a satellite is always in the loop to 
receive and rebroadcast signals from the 
Earth station. • : i v : 
King and'Rosen together teach the use or a 
network of satellites.,, in low. Earth orbit.. 
Thus Ruddy becomes the piece of the prior 
art mosaic that shows, in the reading of the 
Board, the use of > plurality of fan beams 
that are elongate in the travel direction oj tne 
satellite." Ruddy, however, is different from 
the claimed invention in several respects. 
Specifically, the application claims the pro- 
jection of multiple elliptical famshape^ foot- 
prints from the satellite to the ground,<See 
Claim 1, supra, see also Application at 6, 
lines 9-11 ("In addition, in this system, the 
geometrical shape of the beams 12 is 
changed: instead of being circular they are 
now elongate ellipses."). The application s 
written description further - teaches that trie 
invention's fan-shaped satellite beams will 
minimize handovers. See id. at lines 11-16 
("This considerably increases calLdurations 
between handovers."). . . 

In contrast, Ruddy teaches that a ground 
station may use ar single fan-shaped beam to 
transmit to. a satellite in a, unique \Molmya 
orbit The ground station transmits a beam 
into which a series of satellites in Molniya 
orbits will successively enter. At least two 
differences are evident: the application 
teaches projection of multiple beams from a 
satellite to the Earth, while Ruddy teaches 
projection of a single beam from the Earth to 
satellites. Moreover to the extent Ruddy con- 
tains a teaching-about handovers, its teach- 
ings focus on useof the unique Molniya orbit 
to ensure that a satellite always falls ;within 
the beam transmitted by the ground station. 
■ These differences suggest some difficulty 
in showing a prima facie case of obviousness 
The Board, however, specifically found that 
artisans of ordinary skill in this field of art 
would know to shift the frame of reference 
from a ground station, following a satellite to 
a satellite-transmitting to the ground Acr 
cording proper deference to the;Board send- 
ing of a lofty skill-level for ordinary artisans 
in this field, this court discerns no clear error 
in the Board's conclusion that these differ- 
ences would not preclude a finding of obvi- 
ousness. While Ruddy does not expressly 
teach alignment of the fan beam with the 
apparent direction. of the satellite's; motion, 
this court perceives no clear ;error in the 
Board's determination .that Ruddy wqujd 
suggest such an alignment to one of skill, in 
this art. Therefore, the Board did not err in 
finding that the combination of King, Rosen, 
and Ruddy contains .all of the elements 
claimed in Rouffet's application, : - 1 . 



[1] However,.the Board reversibly erred in 
determining that one of skill in the art would 
have been motivated to combine these reler- 
ences in a manner that rendered the claimed 
invention obvious, Indeed/the Board did not 
identify any motivation to choose these reier- 
ences for combination, Ruddy, does not spe- 
cifically address handover minimization.. lo 
the extent that Ruddy at all addresses hand- 
overs due to satellite motion, it addresses; this 
subject through the selection of orbital pa- 
rameters. Ruddy does not teach thechoice.oi 
a particular shape ; : and alignment: .of the 
beam projected by the satellite. Thus Ruddy 
addresses the handover problem with an or- 
bit selection, not a beam .shape. -The; Board 
provides no reasons that one.of ordinary skill; 
in this art, seeking to minimize handovers 
due to satellite motion,, would combine, ^Rud- 
dy with Rosea and .King .in a manner that 
would render the claimed/invention pbvious. 

. Obviousness is determined from the yah- 
. tage point of a hypothetical, person having 
ordinary skill in the art to which the patent 
pertains See 35 U.S.C. § 103(a), This. legal 
construct is akin to the "reasonable person 
used as a reference in negligence determina- 
tions. The legal construct also presumes that 
all prior art .references , in the. field of .tne 
invention are available to this -hyp^hetical 
skilled artisan.. See In re C^^m-f.M 
L032, 1038, 25 USPQ2d 1207, 1211-(Fed : 
Cir. 1993). ..... . -: --^ . .. ■.; -.j 

* As this court has stated,,, virtually all 
[inventions] are combinations of old ^ ele- 
ments." Environmental J> esi S ns ^£ td ^^ 
Union Oil CW713 R2d 693 698, 218 
USPQ 865, =870. (Fed. Gir. l?83);;^e also 
Richdel, Inc. , v, = Sunspool Corp >-,lM F 2d 
1573 1579-80, 219 USPQ 8, 12 (Fed. Cir. 
1983) ("Most, if not all,, inventions are cotn- 
binations and mostly ; ( of old elements. ).. 
Therefore an examiner.may often find ey^ry 
element of a claimed invention in the prior 
art If identification of each claimed, element 
in the prior art were sufficient t.Ojnegate 
patentability, very few patents would. ; ever 
issue: Furthermore, rejecting patent?, solely - 
by finding prior art ; corollaries . Tor the 
claimed elements would permit an examiner 
to use the claimed invention. itself as.a . blue- 
print for piecing together elements in the 
prior art to defeat the. patentability of the 
claimed invention. Such . an approach would 
be "an illogical and inappropriate process by 
which to determine patentability;'! Sjnson- 
ics, Inc. v. Aerosonic Corp., 81 F3d 1566, 
1570, 38 USPQ2d 1551, 1554 (Fed. Cir. 

1996). ' ' . J . . . , 

To prevent the use of hindsight based, on 
the invention to defeat patentability of the 
invention, this court requires the examiner^ 
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show a motivation to combine the references 
that create the case of obviousness. In other 
words, the examiner must show f easoris that 
the skilled artisan, confronted with the same 
problems as the inventor arid with no knowl- 
edge of the Claimed invention, would select 
the- elements from the cited prior art refer- 
ences for ' combination in the mariner 
claimed. ' '* - ft • 
■= [2]' This court has identified three possible 
sources: for a motivation to combine refer- 
ences:' the nature of the problem to be ; solved, 
th¥ teachings bf the prior art, and the- knowl- 
edge of persons of ordinary skill in the art Jin 
this case," the Board relied upon hone of 
these? Rather^- just as "it relied on the .high 
level ;6f skill . Tn the art to overcome the 
differences between the claimed invention 
and the selected elements in the references, it 
relied upon the high level of skill in the art to 
provide the necessary motivation^ The Board 
did not; however, explain what" specific un- 
derstanding or technological principle within 
the knowledge of one of ordinary skill in the 
art would Have suggested the combination: 
Instead; the Board merely invoked the high 
level of skill in the field of art. If such a : rbte ' 
invocation could suffice to supply a motiva : 
tion to combine, the more sophisticated, sci- 
entific fields would rarely; if ever, experience 
a patentable technical advance. Instead, in 
complex scientific fields, the Board could 
routinely identify the prior art elements in an 
application, invoke the lofty level of. skill; and 
rest its case for rejection. To counter' this 
potential weakness in the obviousness con- 
struct, 1 the suggestion to combine require- 
ment stands as a critical safeguard against 
hindsight analysis and rote application of the 
le^al test for obviousness: " : - - . / 1 , 

Because "the -Board did not . explain the 
specific understanding 1 or principle "within 
the knowledge of a skilled artisan that would 
jTintivate one with no knowledge of Roiiffet's 
invention to- make the combination, ; this 
court infers that the examiner selected these 
references with the assistance of hindsight: 
This court forbids the use of hindsight in the 
selection of references that Comprise the case 
of obviousness. See In re Gorman, 933 F.2d 
982, ; 986, 18 USPQ2d 1885, 1888 (Fed. Cir. 
1991). Lacking a ^ motivation to combine 
references, the Board did not show a -proper 
prima facie case of obviousness.- This court 
reverses "the rejection over the combination 
of King, Rosen, arid Ruddy. • 

The Combination of Freeburg and Levine 

Freeburg teaches a ' cellular radiotele- 
phone system based on a constellation of low 
Earth orbit satellites that use conical beams 



to transmit from" the satellite tcr both fixed 
and mobile Earth stations. 1 Levine teaches ah 
Earth-based cellular radio system that uses 
fan beams broadcast from antenna towers: 
Levirie's elliptical footprints are aligned with 
the road 1 grid.To increase the capacity of 
traditional ground-based ; systerris Hhrougri 
frequency reuse techniques^ Levine teaches 
the use of antennas that broadcast signals 
with 'smaller footprints than the prior art- 
system. Thus, Levine actually; increases the 
number of overlap regions between cells arid," 
hence, the number of potential handovers* 
Figure 1 of the Levihe patent illustrates its. 
alignment of bearii footprints:- : '-*' 'y'^l 




•< As a mobile unit (e.g., a driver using a car 
phone) moves though a successions of over- 
lapping zones; Levine uses selection algor 
rithms ttf determine " which, of the cells.is 
aligned with the travel direction of the frio- 
bile.unit. These algorithms then select this 
cell for use while continually monitoring in- 
tersecting cells in: ; the event that the. mobile 
unit changes direction, i: r r ; 

Once again, this court: notes significant 
differences between the teachings of the a r> 
plication and the Levine-Fireeburg corribina; 
tion. The critical Levine reference again in- 
volves a beam from" an Earth station without 
any reference to the "travel direction of [a] 
satellite.'- Moreover, Levine actually multi- 
plies the nuinber of potential handovers and 
then uses software to sort out the necessary 
handovers from the unnecessary. However; 
the Board explains the reasons that one pos; 
sessing the lofty skills characteristic of this 
field would know to account for the differ- 
ences between the claimed invention and the 
prior art combination. This court discerns no 
clear error in that reliance on the consider- 
able skills in this^field: : " 
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• [3} .This -court does, however,- discern re- 
versible error ; in the Board's identification of 
a ; motivation to combine Levine arid Free- 
burg. In : determining that/one of. skill in the 
art would have had motivation to: combine 
Levine and Freeburg, the Board noted that 
"[■t]he revel of skill in the art is very high." 
As rioted before, this; observation': alone can r 
not supply the required suggestion to com- 
bine these references. The Board posits that 
the high level of skill in the art overcomes the 
absence of any actual suggestion that one 
could -select part of the teachings of /Levine 
for combination with the- satellite system 
disclosed 'by; Freeburg. ^ : v. ...:"*. 

- As noted above, the suggestion to combine 
requirement is a. safeguard- against the use of 
hindsight-combinatioriS;to negate patentabil- 
ity. -While the skill level is a component pf 
the inquiry for a suggestion to combine, *a 
lofty level of -skill; alone does "not suffice to 
supply a motivation to combine. Otherwise a 
high level - of ordinary skill in an art afield 
would alhiost always preclude: patentable in- 
ventions: As ! this court has often : noted,- mo- 
vent iom itself is the process 'of combining 
prior >art" : in a -nonobvious rnahner . See, e.g. , 
Richdel; 714 F.2d at 151 9\ Environmental * 
Designs, 113 F.2d at 698. Therefore;- :even 
when" the level of skill in. the art is high, the . 
Board jmust 'identify specifically the princi- 
ple- known* "to : one of ordinary skill, "that 
suggests the '"claimed combination. <i-Gf. 
Gechter'v. Davidson\ 116 F.3d . 1454,^43 
USPQ2d 1-030 (Fed. Gir. 1997) (explaining 
that the Board's opinion must describe the 
basis for its decision). In other .Words, the 
Board -rh Us t explain the reasons one of ordi- 
nary skill in the art would have been motivat- 
ed to' select the references and to combine ' 
them to render : the claimed - invention 
obvious/" ; i'-'"^.-:- 
i The Board's naked invocation :of skill in 
theart toSupply-a suggestion to combine the 
references ; cited iri : this case is.;; therefore 
clearly^erroneous. Absent. any proper motiva- 
tion to combine part of Levine's .teachings 
with Freeburg -sv satellite system, the rejec- 
tion of Rouffetfs' claim over these references 
was improper and is reversed. ... .; ; 

- ' - : : - . iv ; V- " \- - ;: 

The Board reversibly erred in determining 
that there was a motivation to combine ei- 
ther the -teachings of King, Rosen, and Rud- 
dy or of Freeburg and Levine in a imanner 
that would render the claimed invention ob- 
vious. Because this predicate was missing* in 
each case, the Board did not properly: show 
that ..these- references render the claimed in- 
vention obvious: : The^fc^^ this . court re- 



verses' the Board!s decision upholding the 
rejection of RoufTet's claims. In sight .of this 
disposition, RoufTet's pending motion to re- 
mand the case to the Board for further con- 
sideration is deni ed , as .moot. . : ■. - * : . :-- : • . : 

' - COSTS' ■; ' 
Each party shall bear its own costs. 

REVERSED: r ; :^b-' 



U.S. Court of Appeals 
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Champagne Louis Roederer.S.A; y. Delicato 
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■ X--?*- -No- 98-1032 X. : r 
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TRADEMARKS AND UNFAIR TRADE 

practices; 1 ;.; ' -':XXP'ZXX\ 

1. Infringement; conflicts between marks ^ 
Likelihood of confusion Particular 
- ^ ^marks ^ — - Confusion .not . -likely 
• - -(§335.0304.05) :: . - ■: \ r : 

1 Infringement; conflicts . between : marks". 
Tests generally (§335.06) -A^'im' 1 

"Trademark Trial arid Appeal Board cJid 
: riot err ih dismissing opposition proceed 
oh. ground that applicant's "Crystal -Creek". 
maTk for. wine and opposer's ;"Crjs^!',marKs 
, ;fpr : C^ anipagne are dissiniiiar^wrth respect; 5 to 
appearance, sound, significance, arid .com- 
mercial impression,' since 1 board did riot err in 
relying Jsolely on dissimilarity of marks : iri 
evaluating likelihood- "of; cdnfusibn , sirite sin- 
gle factor may be dispbsitive in likehhodd of 
confusion analysis, especially when. that fac- 
tor is dissimilarity of marks, and since ho 
instances of clear error , regarding board's 
.findings of fact as to dissirn'Oarities of marks 
: have been derrionst rated. " ' 



- Appeal from the. U.S. Patent- and Trade- 
mark 7 Office, Trademark Trial :and Appeal 
Board;.'-' ' : ^-- f 

Proceeding (Opposition . No.. . 80,932) 
brought by Champagne Louis:Roede~rerS.A. 
in opposition to application filed by Delicato 
. Vineyards to register mark "Crystal Creek" 
for wine. From dismissal, of opposition pro- 
ceeding, opposer appeals.; Affirmed; Michel, 
J;, concurring in separate opinion; .{■, s 
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